2024-03-28T01:47:19.079 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T01:47:19.079 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} --verbose --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289 --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --owner mchangir -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/orig.config.yaml 2024-03-28T01:47:19.137 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T01:47:19.225 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} email: mchangir@redhat.com first_in_suite: false job_id: '7625289' kernel: client: sha1: distro flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDNAGKdEIkOeJdGKSc6vvM9vhCHIa6CwM6nCt7DBJnb28/FZeHbZP9ERQHmy2FA4MqdvMsXR7fXwHh5j9ytxrDE= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHsnuOuuYtDHnkI1Br+l96AJYXcu28EuymJYWEf70mN7nvEIJGYBgF46BbrQoI+7RYW/vDw6TpVGSNuerNtMcw= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOJYoPZDfxS+Y1BtjGEiDw4H4FGd9SCJqVutLJhWFCSVGatw8gGpssdXLA9drWNqQ0OFhvs3Aq0mc7tUbtlSak= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T01:47:19.225 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T01:47:19.226 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T01:47:19.226 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T01:47:19.229 INFO:teuthology.task.internal:Checking packages... 2024-03-28T01:47:19.254 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T01:47:19.254 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T01:47:19.254 INFO:teuthology.packaging:ref: None 2024-03-28T01:47:19.255 INFO:teuthology.packaging:tag: None 2024-03-28T01:47:19.255 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T01:47:19.255 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:47:19.255 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T01:47:19.403 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T01:47:19.405 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T01:47:19.411 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T01:47:19.412 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T01:47:19.461 INFO:teuthology.task.internal:Saving configuration 2024-03-28T01:47:19.482 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T01:47:19.489 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T01:47:19.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 01:42:03.260581', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDNAGKdEIkOeJdGKSc6vvM9vhCHIa6CwM6nCt7DBJnb28/FZeHbZP9ERQHmy2FA4MqdvMsXR7fXwHh5j9ytxrDE='} 2024-03-28T01:47:19.538 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 01:42:03.257150', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHsnuOuuYtDHnkI1Br+l96AJYXcu28EuymJYWEf70mN7nvEIJGYBgF46BbrQoI+7RYW/vDw6TpVGSNuerNtMcw='} 2024-03-28T01:47:19.565 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 01:42:03.258961', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOJYoPZDfxS+Y1BtjGEiDw4H4FGd9SCJqVutLJhWFCSVGatw8gGpssdXLA9drWNqQ0OFhvs3Aq0mc7tUbtlSak='} 2024-03-28T01:47:19.565 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T01:47:19.573 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T01:47:19.573 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T01:47:19.573 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T01:47:19.573 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T01:47:19.698 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f83312fc0d0>, signals=[15]) 2024-03-28T01:47:19.698 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T01:47:19.707 INFO:teuthology.task.internal:Opening connections... 2024-03-28T01:47:19.707 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T01:47:19.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:47:19.808 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-28T01:47:19.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:47:19.896 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-03-28T01:47:19.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:47:19.983 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T01:47:19.990 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-03-28T01:47:20.008 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-03-28T01:47:20.009 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-03-28T01:47:20.066 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="CentOS Stream" 2024-03-28T01:47:20.066 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="9" 2024-03-28T01:47:20.066 INFO:teuthology.orchestra.run.smithi106.stdout:ID="centos" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="rhel fedora" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="9" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el9" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:LOGO="fedora-logo-icon" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://centos.org/" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T01:47:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T01:47:20.068 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-03-28T01:47:20.102 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-28T01:47:20.120 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-28T01:47:20.120 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-28T01:47:20.178 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-03-28T01:47:20.178 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-03-28T01:47:20.178 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-03-28T01:47:20.178 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-03-28T01:47:20.178 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-03-28T01:47:20.178 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-03-28T01:47:20.178 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T01:47:20.178 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-03-28T01:47:20.178 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-03-28T01:47:20.179 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T01:47:20.179 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-03-28T01:47:20.179 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T01:47:20.179 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T01:47:20.179 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T01:47:20.179 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-28T01:47:20.202 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-03-28T01:47:20.220 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-03-28T01:47:20.220 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-03-28T01:47:20.276 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2024-03-28T01:47:20.276 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="9" 2024-03-28T01:47:20.276 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="9" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el9" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:LOGO="fedora-logo-icon" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T01:47:20.277 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T01:47:20.277 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-03-28T01:47:20.301 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T01:47:20.311 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T01:47:20.336 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T01:47:20.336 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-03-28T01:47:20.338 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-28T01:47:20.341 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-03-28T01:47:20.356 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T01:47:20.378 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T01:47:20.378 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-03-28T01:47:20.395 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-28T01:47:20.397 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-03-28T01:47:20.502 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T01:47:20.579 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T01:47:20.593 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T01:47:20.593 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-28T01:47:20.593 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-28T01:47:20.594 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T01:47:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:47:20.610 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-03-28T01:47:20.669 INFO:teuthology.orchestra.run.smithi106.stdout:5.14.0-361.el9.x86_64 2024-03-28T01:47:20.669 INFO:teuthology.task.kernel:Running kernel on smithi106: 5.14.0-361.el9.x86_64 2024-03-28T01:47:20.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum install -y kernel 2024-03-28T01:47:22.847 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-28T01:47:24.797 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - BaseOS 2.4 MB/s | 2.0 MB 00:00 2024-03-28T01:47:29.037 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-28T01:47:32.415 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 19 MB 00:02 2024-03-28T01:47:43.705 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - CRB 800 kB/s | 6.1 MB 00:07 2024-03-28T01:47:45.063 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-28T01:47:53.888 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 2.4 MB/s | 21 MB 00:08 2024-03-28T01:47:58.917 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-28T01:48:01.082 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T01:48:01.083 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T01:48:01.175 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-03-28T01:48:01.186 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-03-28T01:48:01.186 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout:Install 4 Packages 2024-03-28T01:48:01.187 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:48:01.188 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 76 M 2024-03-28T01:48:01.188 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 123 M 2024-03-28T01:48:01.188 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-03-28T01:48:01.560 INFO:teuthology.orchestra.run.smithi106.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 623 kB/s | 66 kB 00:00 2024-03-28T01:48:01.919 INFO:teuthology.orchestra.run.smithi106.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 32 MB/s | 15 MB 00:00 2024-03-28T01:48:02.394 INFO:teuthology.orchestra.run.smithi106.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 36 MB/s | 33 MB 00:00 2024-03-28T01:48:02.720 INFO:teuthology.orchestra.run.smithi106.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 24 MB/s | 27 MB 00:01 2024-03-28T01:48:02.720 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-03-28T01:48:02.720 INFO:teuthology.orchestra.run.smithi106.stdout:Total 50 MB/s | 76 MB 00:01 2024-03-28T01:48:03.389 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-03-28T01:48:03.755 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-03-28T01:48:03.755 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-03-28T01:48:04.767 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-03-28T01:48:04.768 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-03-28T01:48:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-03-28T01:48:07.829 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T01:48:07.911 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T01:48:10.642 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T01:48:10.782 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T01:48:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T01:48:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T01:48:20.263 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T01:48:49.740 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T01:48:49.741 INFO:teuthology.orchestra.run.smithi106.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T01:48:49.741 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:48:49.784 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T01:48:50.146 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T01:48:50.146 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T01:48:50.146 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T01:48:50.146 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T01:48:50.562 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T01:48:50.562 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:48:50.562 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-03-28T01:48:50.562 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T01:48:50.562 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T01:48:50.562 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T01:48:50.563 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T01:48:50.563 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:48:50.563 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-03-28T01:48:50.892 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T01:48:50.992 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T01:48:50.992 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T01:48:50.992 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T01:48:50.993 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T01:48:50.993 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T01:48:50.993 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-28T01:48:50.993 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi106.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T01:48:50.993 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T01:48:51.073 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T01:48:51.073 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T01:48:51.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo rpm -qi grub2-tools 2024-03-28T01:48:51.108 INFO:teuthology.orchestra.run.smithi106.stdout:Name : grub2-tools 2024-03-28T01:48:51.108 INFO:teuthology.orchestra.run.smithi106.stdout:Epoch : 1 2024-03-28T01:48:51.108 INFO:teuthology.orchestra.run.smithi106.stdout:Version : 2.06 2024-03-28T01:48:51.108 INFO:teuthology.orchestra.run.smithi106.stdout:Release : 68.el9 2024-03-28T01:48:51.108 INFO:teuthology.orchestra.run.smithi106.stdout:Architecture: x86_64 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Group : Unspecified 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Size : 8264254 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:License : GPLv3+ 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Packager : builder@centos.org 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Vendor : CentOS 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Summary : Support tools for GRUB. 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout:Description : 2024-03-28T01:48:51.109 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:48:51.110 INFO:teuthology.orchestra.run.smithi106.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T01:48:51.110 INFO:teuthology.orchestra.run.smithi106.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T01:48:51.110 INFO:teuthology.orchestra.run.smithi106.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T01:48:51.110 INFO:teuthology.orchestra.run.smithi106.stdout:hardware devices. 2024-03-28T01:48:51.110 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:48:51.110 INFO:teuthology.orchestra.run.smithi106.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T01:48:51.111 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T01:48:51.111 INFO:teuthology.task.kernel:Updating grub on smithi106 to boot 5.14.0-432.el9.x86_64 2024-03-28T01:48:51.111 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T01:48:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:Generating grub configuration file ... 2024-03-28T01:48:53.049 INFO:teuthology.orchestra.run.smithi106.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T01:48:53.088 INFO:teuthology.orchestra.run.smithi106.stderr:done 2024-03-28T01:48:53.090 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-03-28T01:48:53.105 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.rur4NEcN72 2024-03-28T01:48:53.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rur4NEcN72 2024-03-28T01:48:53.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.rur4NEcN72 2024-03-28T01:48:53.320 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.rur4NEcN72 is 6KB 2024-03-28T01:48:53.370 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.rur4NEcN72 2024-03-28T01:48:53.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T01:48:53.455 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T01:48:53.455 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T01:48:53.455 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T01:48:53.455 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T01:48:53.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T01:48:53.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo shutdown -r now 2024-03-28T01:49:23.616 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T01:49:23.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-28T01:49:23.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:49:42.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-03-28T01:49:51.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-28T01:49:51.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:49:54.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-03-28T01:50:06.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-28T01:50:06.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:50:24.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-03-28T01:50:39.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-28T01:50:39.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:50:39.909 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-03-28T01:50:40.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-28T01:50:40.281 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-432.el9.x86_64"... 2024-03-28T01:50:40.281 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-03-28T01:50:40.295 INFO:teuthology.orchestra.run.smithi106.stdout:5.14.0-432.el9.x86_64 2024-03-28T01:50:40.295 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T01:50:40.295 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T01:50:40.295 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T01:50:41.296 DEBUG:teuthology.parallel:result is None 2024-03-28T01:50:41.296 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T01:50:41.304 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T01:50:41.304 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T01:50:41.307 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T01:50:41.309 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T01:50:41.328 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T01:50:41.372 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T01:50:41.397 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T01:50:41.397 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T01:50:41.400 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T01:50:41.402 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T01:50:41.536 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T01:50:41.544 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T01:50:41.544 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T01:50:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:50:41.561 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T01:50:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:50:41.577 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T01:50:41.580 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T01:50:41.604 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T01:50:41.639 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T01:50:41.652 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T01:50:41.661 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T01:50:41.662 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T01:50:41.686 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T01:50:41.708 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T01:50:41.710 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T01:50:41.718 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T01:50:41.719 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T01:50:41.721 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T01:50:41.723 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T01:50:41.753 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T01:50:41.763 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T01:50:41.763 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T01:50:41.790 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T01:50:41.794 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T01:50:41.809 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T01:50:41.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T01:50:42.094 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T01:50:42.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T01:50:42.180 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T01:50:42.180 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T01:50:42.248 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T01:50:42.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T01:50:42.368 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T01:50:42.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T01:50:42.455 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T01:50:42.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T01:50:42.524 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T01:50:42.566 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T01:50:42.623 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T01:50:42.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T01:50:42.752 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T01:50:42.752 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T01:50:42.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-03-28T01:50:42.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-28T01:50:42.827 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-03-28T01:50:42.866 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T01:50:42.875 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T01:50:42.901 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T01:50:43.401 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T01:50:43.411 INFO:teuthology.task.internal:Starting timer... 2024-03-28T01:50:43.411 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T01:50:43.453 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T01:50:43.481 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T01:50:43.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo service auditd rotate 2024-03-28T01:50:43.568 INFO:teuthology.orchestra.run.smithi106.stdout:Rotating logs: 2024-03-28T01:50:43.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-03-28T01:50:43.632 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-03-28T01:50:43.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-03-28T01:50:43.695 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-03-28T01:50:43.696 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T01:50:43.697 DEBUG:teuthology.orchestra.run.smithi106:> /usr/sbin/getenforce 2024-03-28T01:50:43.717 INFO:teuthology.orchestra.run.smithi106.stdout:Permissive 2024-03-28T01:50:43.718 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-03-28T01:50:43.755 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-03-28T01:50:43.756 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-03-28T01:50:43.775 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-03-28T01:50:43.775 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi106.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-03-28T01:50:43.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T01:50:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:50:43.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T01:50:43.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:50:43.835 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T01:50:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:50:43.863 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T01:50:43.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo /usr/sbin/setenforce permissive 2024-03-28T01:50:43.894 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-03-28T01:50:43.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-03-28T01:50:43.969 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T01:50:43.979 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-28T01:50:44.015 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-28T01:50:44.371 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T01:50:44.387 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T01:50:44.388 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi106.front.sepia.ceph.com,smithi153.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T01:57:06.112 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi106.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-03-28T01:57:06.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-28T01:57:06.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:57:06.205 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-03-28T01:57:06.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-28T01:57:06.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-28T01:57:06.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:57:06.370 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-28T01:57:06.438 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-28T01:57:06.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-28T01:57:06.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T01:57:06.523 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-03-28T01:57:06.589 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-28T01:57:06.589 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T01:57:06.600 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T01:57:06.600 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T01:57:06.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T01:57:06.603 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T01:57:06.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T01:57:06.605 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T01:57:06.605 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T01:57:06.631 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T01:57:06.635 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T01:57:06.648 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T01:57:06.652 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T01:57:06.659 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T01:57:06.675 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T01:57:06.677 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: ntpd: command not found 2024-03-28T01:57:06.682 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-03-28T01:57:06.690 INFO:teuthology.orchestra.run.smithi106.stdout:506 Cannot talk to daemon 2024-03-28T01:57:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-03-28T01:57:06.704 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-03-28T01:57:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T01:57:06.712 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T01:57:06.718 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-03-28T01:57:06.723 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T01:57:06.728 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T01:57:06.734 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T01:57:06.750 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T01:57:06.772 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-28T01:57:06.776 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T01:57:06.776 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-28T01:57:06.776 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.777 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.777 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.777 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.777 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-03-28T01:57:06.780 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T01:57:06.780 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-03-28T01:57:06.780 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.780 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.780 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.780 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.793 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-03-28T01:57:06.796 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T01:57:06.796 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-03-28T01:57:06.797 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.797 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.797 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.797 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:57:06.798 INFO:teuthology.run_tasks:Running task install... 2024-03-28T01:57:06.805 DEBUG:teuthology.task.install:project ceph 2024-03-28T01:57:06.805 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T01:57:06.806 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T01:57:06.806 INFO:teuthology.task.install:Using flavor: default 2024-03-28T01:57:06.813 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T01:57:06.813 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T01:57:06.813 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T01:57:06.814 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:57:06.816 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T01:57:06.816 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:57:06.817 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T01:57:06.817 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:57:06.962 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T01:57:06.962 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T01:57:07.031 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T01:57:07.031 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T01:57:07.046 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T01:57:07.046 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T01:57:07.134 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T01:57:07.134 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T01:57:07.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T01:57:07.136 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T01:57:07.136 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T01:57:07.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T01:57:07.153 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T01:57:07.153 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T01:57:07.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T01:57:07.163 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T01:57:07.164 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T01:57:07.165 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T01:57:07.165 DEBUG:teuthology.orchestra.run.smithi106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T01:57:07.181 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T01:57:07.181 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T01:57:07.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T01:57:07.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T01:57:07.249 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T01:57:07.312 DEBUG:teuthology.orchestra.run.smithi106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T01:57:07.312 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T01:57:07.328 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T01:57:07.378 INFO:teuthology.orchestra.run.smithi106.stdout:check_obsoletes = 1 2024-03-28T01:57:07.378 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-03-28T01:57:07.379 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum clean all 2024-03-28T01:57:07.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-03-28T01:57:07.395 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-03-28T01:57:07.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-03-28T01:57:07.616 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-03-28T01:57:07.629 INFO:teuthology.orchestra.run.smithi178.stdout:74 files removed 2024-03-28T01:57:07.643 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T01:57:07.646 INFO:teuthology.orchestra.run.smithi106.stdout:74 files removed 2024-03-28T01:57:07.656 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T01:57:07.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T01:57:08.257 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 256 kB/s | 83 kB 00:00 2024-03-28T01:57:08.325 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 218 kB/s | 83 kB 00:00 2024-03-28T01:57:08.334 INFO:teuthology.orchestra.run.smithi106.stdout:ceph packages for x86_64 231 kB/s | 83 kB 00:00 2024-03-28T01:57:08.632 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-28T01:57:08.651 INFO:teuthology.orchestra.run.smithi106.stdout:ceph noarch packages 38 kB/s | 10 kB 00:00 2024-03-28T01:57:08.668 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-03-28T01:57:08.983 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-28T01:57:09.060 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-28T01:57:09.068 INFO:teuthology.orchestra.run.smithi106.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-28T01:57:09.496 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-28T01:57:09.554 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-28T01:57:09.644 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-28T01:57:10.996 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-28T01:57:10.997 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-28T01:57:11.128 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-28T01:57:12.306 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-03-28T01:57:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-03-28T01:57:14.855 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-28T01:57:18.789 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-28T01:57:20.298 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-28T01:57:20.411 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - BaseOS 907 kB/s | 8.0 MB 00:09 2024-03-28T01:57:22.337 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-28T01:57:22.770 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-28T01:57:26.688 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-28T01:57:27.242 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-28T01:57:28.339 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-28T01:57:29.603 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T01:57:29.604 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T01:57:29.616 INFO:teuthology.orchestra.run.smithi178.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T01:57:29.617 INFO:teuthology.orchestra.run.smithi178.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T01:57:29.618 INFO:teuthology.orchestra.run.smithi178.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T01:57:29.619 INFO:teuthology.orchestra.run.smithi178.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T01:57:29.621 INFO:teuthology.orchestra.run.smithi178.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T01:57:29.624 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T01:57:29.624 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T01:57:29.704 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-28T01:57:29.722 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-28T01:57:29.722 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-28T01:57:29.722 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-28T01:57:29.722 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-28T01:57:29.722 INFO:teuthology.orchestra.run.smithi178.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T01:57:29.722 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T01:57:29.722 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T01:57:29.723 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T01:57:29.724 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T01:57:29.725 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T01:57:29.726 INFO:teuthology.orchestra.run.smithi178.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T01:57:29.727 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T01:57:29.728 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T01:57:29.729 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T01:57:29.730 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T01:57:29.731 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T01:57:29.732 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T01:57:29.733 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T01:57:29.734 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T01:57:29.735 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T01:57:29.736 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T01:57:29.737 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout:Install 150 Packages 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 24 Packages 2024-03-28T01:57:29.738 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:57:29.739 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 225 M 2024-03-28T01:57:29.739 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-28T01:57:31.143 INFO:teuthology.orchestra.run.smithi178.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 34 kB/s | 6.6 kB 00:00 2024-03-28T01:57:31.460 INFO:teuthology.orchestra.run.smithi178.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 10 MB/s | 5.4 MB 00:00 2024-03-28T01:57:31.493 INFO:teuthology.orchestra.run.smithi178.stdout:(3/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.5 MB/s | 890 kB 00:00 2024-03-28T01:57:31.518 INFO:teuthology.orchestra.run.smithi178.stdout:(4/174): ceph-immutable-object-cache-19.0.0-214 2.4 MB/s | 145 kB 00:00 2024-03-28T01:57:31.627 INFO:teuthology.orchestra.run.smithi178.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 14 MB/s | 1.5 MB 00:00 2024-03-28T01:57:31.902 INFO:teuthology.orchestra.run.smithi178.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 17 MB/s | 4.7 MB 00:00 2024-03-28T01:57:31.945 INFO:teuthology.orchestra.run.smithi178.stdout:(7/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 5.1 MB/s | 2.3 MB 00:00 2024-03-28T01:57:32.787 INFO:teuthology.orchestra.run.smithi178.stdout:(8/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 19 MB/s | 17 MB 00:00 2024-03-28T01:57:32.820 INFO:teuthology.orchestra.run.smithi178.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 761 kB/s | 25 kB 00:00 2024-03-28T01:57:34.029 INFO:teuthology.orchestra.run.smithi178.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 4.7 MB/s | 9.9 MB 00:02 2024-03-28T01:57:34.096 INFO:teuthology.orchestra.run.smithi178.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.2 MB/s | 220 kB 00:00 2024-03-28T01:57:34.187 INFO:teuthology.orchestra.run.smithi178.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 295 kB/s | 27 kB 00:00 2024-03-28T01:57:34.280 INFO:teuthology.orchestra.run.smithi178.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 341 kB/s | 31 kB 00:00 2024-03-28T01:57:34.530 INFO:teuthology.orchestra.run.smithi178.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 2.9 MB/s | 746 kB 00:00 2024-03-28T01:57:34.630 INFO:teuthology.orchestra.run.smithi178.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.6 MB/s | 163 kB 00:00 2024-03-28T01:57:34.722 INFO:teuthology.orchestra.run.smithi178.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-28T01:57:35.222 INFO:teuthology.orchestra.run.smithi178.stdout:(17/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 20 MB/s | 48 MB 00:02 2024-03-28T01:57:35.256 INFO:teuthology.orchestra.run.smithi178.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 939 kB/s | 501 kB 00:00 2024-03-28T01:57:35.275 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 889 kB/s | 19 MB 00:21 2024-03-28T01:57:35.331 INFO:teuthology.orchestra.run.smithi178.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 602 kB/s | 45 kB 00:00 2024-03-28T01:57:35.423 INFO:teuthology.orchestra.run.smithi178.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.4 MB/s | 131 kB 00:00 2024-03-28T01:57:35.498 INFO:teuthology.orchestra.run.smithi178.stdout:(21/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 19 MB/s | 5.1 MB 00:00 2024-03-28T01:57:35.524 INFO:teuthology.orchestra.run.smithi178.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.6 MB/s | 162 kB 00:00 2024-03-28T01:57:35.549 INFO:teuthology.orchestra.run.smithi178.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 6.2 MB/s | 321 kB 00:00 2024-03-28T01:57:35.574 INFO:teuthology.orchestra.run.smithi178.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 5.8 MB/s | 298 kB 00:00 2024-03-28T01:57:35.600 INFO:teuthology.orchestra.run.smithi178.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.9 MB/s | 99 kB 00:00 2024-03-28T01:57:35.625 INFO:teuthology.orchestra.run.smithi178.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-28T01:57:35.726 INFO:teuthology.orchestra.run.smithi178.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.7 MB/s | 170 kB 00:00 2024-03-28T01:57:35.784 INFO:teuthology.orchestra.run.smithi178.stdout:(28/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 17 MB/s | 3.1 MB 00:00 2024-03-28T01:57:35.809 INFO:teuthology.orchestra.run.smithi178.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2145.g 406 kB/s | 34 kB 00:00 2024-03-28T01:57:35.835 INFO:teuthology.orchestra.run.smithi178.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.7 MB/s | 139 kB 00:00 2024-03-28T01:57:36.102 INFO:teuthology.orchestra.run.smithi178.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 6.2 MB/s | 1.8 MB 00:00 2024-03-28T01:57:36.177 INFO:teuthology.orchestra.run.smithi178.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.2 MB/s | 248 kB 00:00 2024-03-28T01:57:36.269 INFO:teuthology.orchestra.run.smithi178.stdout:(33/174): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-03-28T01:57:36.294 INFO:teuthology.orchestra.run.smithi178.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 421 kB/s | 49 kB 00:00 2024-03-28T01:57:36.320 INFO:teuthology.orchestra.run.smithi178.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 329 kB/s | 16 kB 00:00 2024-03-28T01:57:36.354 INFO:teuthology.orchestra.run.smithi178.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 264 kB 00:00 2024-03-28T01:57:36.387 INFO:teuthology.orchestra.run.smithi178.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 11 MB/s | 748 kB 00:00 2024-03-28T01:57:36.413 INFO:teuthology.orchestra.run.smithi178.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 312 kB/s | 18 kB 00:00 2024-03-28T01:57:36.570 INFO:teuthology.orchestra.run.smithi178.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2024-03-28T01:57:36.620 INFO:teuthology.orchestra.run.smithi178.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.3 MB/s | 920 kB 00:00 2024-03-28T01:57:36.645 INFO:teuthology.orchestra.run.smithi178.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2024-03-28T01:57:36.671 INFO:teuthology.orchestra.run.smithi178.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.9 MB/s | 307 kB 00:00 2024-03-28T01:57:36.696 INFO:teuthology.orchestra.run.smithi178.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 631 kB/s | 32 kB 00:00 2024-03-28T01:57:36.780 INFO:teuthology.orchestra.run.smithi178.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 27 MB/s | 3.0 MB 00:00 2024-03-28T01:57:36.806 INFO:teuthology.orchestra.run.smithi178.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 160 kB/s | 17 kB 00:00 2024-03-28T01:57:36.831 INFO:teuthology.orchestra.run.smithi178.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-28T01:57:36.856 INFO:teuthology.orchestra.run.smithi178.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-28T01:57:36.882 INFO:teuthology.orchestra.run.smithi178.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-28T01:57:36.915 INFO:teuthology.orchestra.run.smithi178.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-28T01:57:36.941 INFO:teuthology.orchestra.run.smithi178.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-28T01:57:36.958 INFO:teuthology.orchestra.run.smithi178.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-28T01:57:36.983 INFO:teuthology.orchestra.run.smithi178.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-28T01:57:37.009 INFO:teuthology.orchestra.run.smithi178.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-28T01:57:37.034 INFO:teuthology.orchestra.run.smithi178.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-28T01:57:37.059 INFO:teuthology.orchestra.run.smithi178.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-28T01:57:37.201 INFO:teuthology.orchestra.run.smithi178.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-28T01:57:37.235 INFO:teuthology.orchestra.run.smithi178.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-28T01:57:37.260 INFO:teuthology.orchestra.run.smithi178.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T01:57:37.455 INFO:teuthology.orchestra.run.smithi178.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-28T01:57:37.477 INFO:teuthology.orchestra.run.smithi178.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 113 kB/s | 24 kB 00:00 2024-03-28T01:57:37.502 INFO:teuthology.orchestra.run.smithi178.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 379 kB/s | 18 kB 00:00 2024-03-28T01:57:37.528 INFO:teuthology.orchestra.run.smithi178.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-28T01:57:37.553 INFO:teuthology.orchestra.run.smithi178.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 629 kB/s | 32 kB 00:00 2024-03-28T01:57:37.695 INFO:teuthology.orchestra.run.smithi178.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-28T01:57:37.720 INFO:teuthology.orchestra.run.smithi178.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-28T01:57:37.746 INFO:teuthology.orchestra.run.smithi178.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-28T01:57:37.771 INFO:teuthology.orchestra.run.smithi178.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T01:57:37.797 INFO:teuthology.orchestra.run.smithi178.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 818 kB/s | 41 kB 00:00 2024-03-28T01:57:37.822 INFO:teuthology.orchestra.run.smithi178.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-28T01:57:37.855 INFO:teuthology.orchestra.run.smithi178.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-28T01:57:37.881 INFO:teuthology.orchestra.run.smithi178.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-28T01:57:37.906 INFO:teuthology.orchestra.run.smithi178.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 941 kB/s | 47 kB 00:00 2024-03-28T01:57:37.932 INFO:teuthology.orchestra.run.smithi178.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 855 kB/s | 43 kB 00:00 2024-03-28T01:57:38.040 INFO:teuthology.orchestra.run.smithi178.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 360 kB/s | 39 kB 00:00 2024-03-28T01:57:38.241 INFO:teuthology.orchestra.run.smithi178.stdout:(75/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 207 kB/s | 41 kB 00:00 2024-03-28T01:57:38.399 INFO:teuthology.orchestra.run.smithi178.stdout:(76/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-28T01:57:38.424 INFO:teuthology.orchestra.run.smithi178.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 300 kB 00:00 2024-03-28T01:57:38.450 INFO:teuthology.orchestra.run.smithi178.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T01:57:38.475 INFO:teuthology.orchestra.run.smithi178.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T01:57:38.500 INFO:teuthology.orchestra.run.smithi178.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-28T01:57:38.526 INFO:teuthology.orchestra.run.smithi178.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T01:57:38.559 INFO:teuthology.orchestra.run.smithi178.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-28T01:57:38.601 INFO:teuthology.orchestra.run.smithi178.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-28T01:57:38.635 INFO:teuthology.orchestra.run.smithi178.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-28T01:57:38.661 INFO:teuthology.orchestra.run.smithi178.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-28T01:57:38.686 INFO:teuthology.orchestra.run.smithi178.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-28T01:57:38.920 INFO:teuthology.orchestra.run.smithi178.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 725 kB/s | 187 kB 00:00 2024-03-28T01:57:38.958 INFO:teuthology.orchestra.run.smithi178.stdout:(88/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-03-28T01:57:39.000 INFO:teuthology.orchestra.run.smithi178.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 195 kB 00:00 2024-03-28T01:57:39.034 INFO:teuthology.orchestra.run.smithi178.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-28T01:57:39.093 INFO:teuthology.orchestra.run.smithi178.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-03-28T01:57:39.118 INFO:teuthology.orchestra.run.smithi178.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-03-28T01:57:39.260 INFO:teuthology.orchestra.run.smithi178.stdout:(93/174): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-28T01:57:39.300 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-28T01:57:39.327 INFO:teuthology.orchestra.run.smithi178.stdout:(94/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2024-03-28T01:57:39.369 INFO:teuthology.orchestra.run.smithi178.stdout:(95/174): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-03-28T01:57:39.394 INFO:teuthology.orchestra.run.smithi178.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-03-28T01:57:39.436 INFO:teuthology.orchestra.run.smithi178.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 5.7 MB/s | 245 kB 00:00 2024-03-28T01:57:39.520 INFO:teuthology.orchestra.run.smithi178.stdout:(98/174): postgresql-server-13.14-1.el9.x86_64. 38 MB/s | 5.8 MB 00:00 2024-03-28T01:57:39.545 INFO:teuthology.orchestra.run.smithi178.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-28T01:57:39.571 INFO:teuthology.orchestra.run.smithi178.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-03-28T01:57:39.596 INFO:teuthology.orchestra.run.smithi178.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 717 kB/s | 36 kB 00:00 2024-03-28T01:57:39.621 INFO:teuthology.orchestra.run.smithi178.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 939 kB/s | 47 kB 00:00 2024-03-28T01:57:39.655 INFO:teuthology.orchestra.run.smithi178.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-28T01:57:39.739 INFO:teuthology.orchestra.run.smithi178.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-03-28T01:57:39.889 INFO:teuthology.orchestra.run.smithi178.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-03-28T01:57:41.098 INFO:teuthology.orchestra.run.smithi178.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 403 kB/s | 548 kB 00:01 2024-03-28T01:57:41.156 INFO:teuthology.orchestra.run.smithi178.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 335 kB/s | 19 kB 00:00 2024-03-28T01:57:41.473 INFO:teuthology.orchestra.run.smithi178.stdout:(108/174): ceph-common-19.0.0-2145.g3be9a5c9.el 1.9 MB/s | 20 MB 00:10 2024-03-28T01:57:41.648 INFO:teuthology.orchestra.run.smithi178.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 149 kB/s | 26 kB 00:00 2024-03-28T01:57:41.740 INFO:teuthology.orchestra.run.smithi178.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 531 kB/s | 49 kB 00:00 2024-03-28T01:57:41.801 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-03-28T01:57:41.809 INFO:teuthology.orchestra.run.smithi178.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 977 kB/s | 67 kB 00:00 2024-03-28T01:57:41.960 INFO:teuthology.orchestra.run.smithi178.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-28T01:57:42.051 INFO:teuthology.orchestra.run.smithi178.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-03-28T01:57:42.177 INFO:teuthology.orchestra.run.smithi178.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-03-28T01:57:42.252 INFO:teuthology.orchestra.run.smithi178.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-03-28T01:57:42.269 INFO:teuthology.orchestra.run.smithi178.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-03-28T01:57:42.302 INFO:teuthology.orchestra.run.smithi178.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-28T01:57:42.319 INFO:teuthology.orchestra.run.smithi178.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-03-28T01:57:42.378 INFO:teuthology.orchestra.run.smithi178.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-28T01:57:42.462 INFO:teuthology.orchestra.run.smithi178.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-03-28T01:57:42.487 INFO:teuthology.orchestra.run.smithi178.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2024-03-28T01:57:42.521 INFO:teuthology.orchestra.run.smithi178.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 604 kB/s | 35 kB 00:00 2024-03-28T01:57:42.663 INFO:teuthology.orchestra.run.smithi178.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.1 MB/s | 203 kB 00:00 2024-03-28T01:57:42.781 INFO:teuthology.orchestra.run.smithi178.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-28T01:57:42.888 INFO:teuthology.orchestra.run.smithi178.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 5.5 MB/s | 2.0 MB 00:00 2024-03-28T01:57:42.914 INFO:teuthology.orchestra.run.smithi178.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 81 kB/s | 11 kB 00:00 2024-03-28T01:57:42.955 INFO:teuthology.orchestra.run.smithi178.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 265 kB/s | 18 kB 00:00 2024-03-28T01:57:42.981 INFO:teuthology.orchestra.run.smithi178.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 345 kB/s | 23 kB 00:00 2024-03-28T01:57:43.031 INFO:teuthology.orchestra.run.smithi178.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-28T01:57:43.056 INFO:teuthology.orchestra.run.smithi178.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-03-28T01:57:43.156 INFO:teuthology.orchestra.run.smithi178.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 464 kB/s | 46 kB 00:00 2024-03-28T01:57:43.223 INFO:teuthology.orchestra.run.smithi178.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-03-28T01:57:43.257 INFO:teuthology.orchestra.run.smithi178.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 786 kB/s | 79 kB 00:00 2024-03-28T01:57:43.324 INFO:teuthology.orchestra.run.smithi178.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-28T01:57:43.424 INFO:teuthology.orchestra.run.smithi178.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-03-28T01:57:43.458 INFO:teuthology.orchestra.run.smithi178.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-28T01:57:43.525 INFO:teuthology.orchestra.run.smithi178.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 897 kB/s | 90 kB 00:00 2024-03-28T01:57:43.558 INFO:teuthology.orchestra.run.smithi178.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-03-28T01:57:43.625 INFO:teuthology.orchestra.run.smithi178.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-03-28T01:57:43.667 INFO:teuthology.orchestra.run.smithi178.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-28T01:57:43.709 INFO:teuthology.orchestra.run.smithi178.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-03-28T01:57:43.751 INFO:teuthology.orchestra.run.smithi178.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-03-28T01:57:43.793 INFO:teuthology.orchestra.run.smithi178.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 610 kB/s | 51 kB 00:00 2024-03-28T01:57:43.877 INFO:teuthology.orchestra.run.smithi178.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-28T01:57:43.977 INFO:teuthology.orchestra.run.smithi178.stdout:(145/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-28T01:57:44.052 INFO:teuthology.orchestra.run.smithi178.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 267 kB/s | 20 kB 00:00 2024-03-28T01:57:44.161 INFO:teuthology.orchestra.run.smithi178.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-28T01:57:44.211 INFO:teuthology.orchestra.run.smithi178.stdout:(148/174): python3-webob-1.8.7-6.el9.noarch.rpm 499 kB/s | 230 kB 00:00 2024-03-28T01:57:44.520 INFO:teuthology.orchestra.run.smithi178.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-03-28T01:57:45.170 INFO:teuthology.orchestra.run.smithi178.stdout:(150/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 4.9 MB/s | 3.1 MB 00:00 2024-03-28T01:57:45.270 INFO:teuthology.orchestra.run.smithi178.stdout:(151/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.2 MB/s | 3.4 MB 00:01 2024-03-28T01:57:45.404 INFO:teuthology.orchestra.run.smithi178.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.3 MB/s | 321 kB 00:00 2024-03-28T01:57:45.671 INFO:teuthology.orchestra.run.smithi178.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 5.8 MB/s | 2.3 MB 00:00 2024-03-28T01:57:45.737 INFO:teuthology.orchestra.run.smithi178.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 632 kB/s | 42 kB 00:00 2024-03-28T01:57:45.821 INFO:teuthology.orchestra.run.smithi178.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-28T01:57:45.888 INFO:teuthology.orchestra.run.smithi178.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-28T01:57:45.955 INFO:teuthology.orchestra.run.smithi178.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 546 kB/s | 36 kB 00:00 2024-03-28T01:57:45.997 INFO:teuthology.orchestra.run.smithi178.stdout:(158/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.1 MB/s | 1.8 MB 00:00 2024-03-28T01:57:46.047 INFO:teuthology.orchestra.run.smithi178.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-28T01:57:46.122 INFO:teuthology.orchestra.run.smithi178.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-28T01:57:46.148 INFO:teuthology.orchestra.run.smithi178.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 230 kB/s | 23 kB 00:00 2024-03-28T01:57:46.198 INFO:teuthology.orchestra.run.smithi178.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-28T01:57:46.224 INFO:teuthology.orchestra.run.smithi178.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-28T01:57:46.273 INFO:teuthology.orchestra.run.smithi178.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-28T01:57:46.299 INFO:teuthology.orchestra.run.smithi178.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 849 kB/s | 64 kB 00:00 2024-03-28T01:57:46.349 INFO:teuthology.orchestra.run.smithi178.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 382 kB/s | 29 kB 00:00 2024-03-28T01:57:46.574 INFO:teuthology.orchestra.run.smithi178.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.6 MB/s | 2.2 MB 00:00 2024-03-28T01:57:46.691 INFO:teuthology.orchestra.run.smithi178.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 3.1 MB/s | 1.2 MB 00:00 2024-03-28T01:57:46.775 INFO:teuthology.orchestra.run.smithi178.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-28T01:57:46.892 INFO:teuthology.orchestra.run.smithi178.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 267 kB/s | 31 kB 00:00 2024-03-28T01:57:46.926 INFO:teuthology.orchestra.run.smithi178.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 1.9 MB/s | 465 kB 00:00 2024-03-28T01:57:46.992 INFO:teuthology.orchestra.run.smithi178.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-03-28T01:57:47.034 INFO:teuthology.orchestra.run.smithi178.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 686 kB/s | 74 kB 00:00 2024-03-28T01:57:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 799 kB/s | 21 MB 00:26 2024-03-28T01:57:55.047 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 1.6 MB/s | 21 MB 00:13 2024-03-28T01:57:55.783 INFO:teuthology.orchestra.run.smithi178.stdout:(174/174): gperftools-libs-2.9.1-2.el9.x86_64.r 19 kB/s | 309 kB 00:15 2024-03-28T01:57:55.789 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-28T01:57:55.789 INFO:teuthology.orchestra.run.smithi178.stdout:Total 8.6 MB/s | 225 MB 00:26 2024-03-28T01:57:56.376 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-28T01:57:56.647 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-28T01:57:56.647 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-28T01:57:58.371 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-28T01:57:58.371 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-28T01:57:59.868 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-28T01:57:59.932 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-28T01:58:02.225 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T01:58:02.226 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T01:58:02.240 INFO:teuthology.orchestra.run.smithi153.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T01:58:02.240 INFO:teuthology.orchestra.run.smithi153.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T01:58:02.241 INFO:teuthology.orchestra.run.smithi153.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T01:58:02.243 INFO:teuthology.orchestra.run.smithi153.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T01:58:02.245 INFO:teuthology.orchestra.run.smithi153.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T01:58:02.248 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T01:58:02.249 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T01:58:02.328 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-28T01:58:02.331 INFO:teuthology.orchestra.run.smithi106.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T01:58:02.332 INFO:teuthology.orchestra.run.smithi106.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T01:58:02.346 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-28T01:58:02.346 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-28T01:58:02.346 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-28T01:58:02.346 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-28T01:58:02.346 INFO:teuthology.orchestra.run.smithi153.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T01:58:02.346 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T01:58:02.346 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T01:58:02.346 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T01:58:02.347 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T01:58:02.348 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T01:58:02.349 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T01:58:02.350 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T01:58:02.351 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T01:58:02.352 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T01:58:02.353 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T01:58:02.354 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T01:58:02.355 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T01:58:02.356 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T01:58:02.357 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T01:58:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-03-28T01:58:02.359 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T01:58:02.360 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:58:02.360 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-28T01:58:02.360 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-28T01:58:02.360 INFO:teuthology.orchestra.run.smithi153.stdout:Install 150 Packages 2024-03-28T01:58:02.360 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 24 Packages 2024-03-28T01:58:02.360 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:58:02.361 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 225 M 2024-03-28T01:58:02.361 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-28T01:58:02.361 INFO:teuthology.orchestra.run.smithi106.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T01:58:02.361 INFO:teuthology.orchestra.run.smithi106.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T01:58:02.361 INFO:teuthology.orchestra.run.smithi106.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T01:58:02.361 INFO:teuthology.orchestra.run.smithi106.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T01:58:02.362 INFO:teuthology.orchestra.run.smithi106.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T01:58:02.362 INFO:teuthology.orchestra.run.smithi106.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T01:58:02.362 INFO:teuthology.orchestra.run.smithi106.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T01:58:02.437 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T01:58:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T01:58:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T01:58:02.460 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrading: 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T01:58:02.461 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T01:58:02.462 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T01:58:02.463 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T01:58:02.464 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T01:58:02.465 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T01:58:02.466 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T01:58:02.467 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T01:58:02.468 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T01:58:02.469 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout:Installing weak dependencies: 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout:Install 150 Packages 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrade 24 Packages 2024-03-28T01:58:02.470 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:58:02.471 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 225 M 2024-03-28T01:58:02.471 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-03-28T01:58:02.978 INFO:teuthology.orchestra.run.smithi153.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-03-28T01:58:03.525 INFO:teuthology.orchestra.run.smithi106.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-03-28T01:58:03.636 INFO:teuthology.orchestra.run.smithi153.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 890 kB 00:00 2024-03-28T01:58:03.753 INFO:teuthology.orchestra.run.smithi153.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.2 MB/s | 145 kB 00:00 2024-03-28T01:58:04.008 INFO:teuthology.orchestra.run.smithi106.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.8 MB/s | 890 kB 00:00 2024-03-28T01:58:04.125 INFO:teuthology.orchestra.run.smithi106.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.2 MB/s | 145 kB 00:00 2024-03-28T01:58:04.617 INFO:teuthology.orchestra.run.smithi106.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 3.8 MB/s | 5.4 MB 00:01 2024-03-28T01:58:04.876 INFO:teuthology.orchestra.run.smithi106.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-03-28T01:58:05.026 INFO:teuthology.orchestra.run.smithi106.stdout:(6/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 2.6 MB/s | 2.3 MB 00:00 2024-03-28T01:58:05.145 INFO:teuthology.orchestra.run.smithi153.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 1.7 MB/s | 2.3 MB 00:01 2024-03-28T01:58:05.457 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-28T01:58:05.582 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T01:58:05.610 INFO:teuthology.orchestra.run.smithi106.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 6.4 MB/s | 4.7 MB 00:00 2024-03-28T01:58:05.621 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T01:58:05.652 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T01:58:05.661 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T01:58:05.701 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T01:58:05.703 INFO:teuthology.orchestra.run.smithi153.stdout:(5/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 6.9 MB/s | 20 MB 00:02 2024-03-28T01:58:05.763 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T01:58:05.781 INFO:teuthology.orchestra.run.smithi153.stdout:(6/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 1.8 MB/s | 5.4 MB 00:03 2024-03-28T01:58:05.797 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T01:58:05.822 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T01:58:05.864 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T01:58:05.871 INFO:teuthology.orchestra.run.smithi153.stdout:(7/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 2.1 MB/s | 1.5 MB 00:00 2024-03-28T01:58:06.194 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T01:58:06.249 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T01:58:06.355 INFO:teuthology.orchestra.run.smithi153.stdout:(8/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 7.2 MB/s | 4.7 MB 00:00 2024-03-28T01:58:06.374 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T01:58:06.400 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T01:58:06.405 INFO:teuthology.orchestra.run.smithi153.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 507 kB/s | 25 kB 00:00 2024-03-28T01:58:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T01:58:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T01:58:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T01:58:06.749 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T01:58:06.802 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T01:58:06.830 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T01:58:06.918 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T01:58:06.947 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T01:58:07.011 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T01:58:07.051 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T01:58:07.074 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T01:58:07.090 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T01:58:07.283 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T01:58:07.313 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T01:58:07.353 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T01:58:07.458 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T01:58:07.556 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T01:58:07.599 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T01:58:07.673 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T01:58:07.685 INFO:teuthology.orchestra.run.smithi106.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 4.7 MB/s | 9.9 MB 00:02 2024-03-28T01:58:07.734 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T01:58:07.768 INFO:teuthology.orchestra.run.smithi106.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 302 kB/s | 25 kB 00:00 2024-03-28T01:58:07.939 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T01:58:08.046 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T01:58:08.142 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T01:58:08.180 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T01:58:08.206 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T01:58:08.242 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T01:58:08.316 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T01:58:08.370 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T01:58:08.409 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T01:58:08.481 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T01:58:08.524 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T01:58:08.568 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T01:58:08.601 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T01:58:08.629 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T01:58:08.697 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T01:58:08.733 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T01:58:08.755 INFO:teuthology.orchestra.run.smithi153.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 3.4 MB/s | 9.9 MB 00:02 2024-03-28T01:58:08.777 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T01:58:08.805 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T01:58:08.875 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T01:58:08.914 INFO:teuthology.orchestra.run.smithi153.stdout:(11/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 5.4 MB/s | 17 MB 00:03 2024-03-28T01:58:08.928 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T01:58:08.939 INFO:teuthology.orchestra.run.smithi153.stdout:(12/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.2 MB/s | 220 kB 00:00 2024-03-28T01:58:08.973 INFO:teuthology.orchestra.run.smithi153.stdout:(13/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 462 kB/s | 27 kB 00:00 2024-03-28T01:58:09.006 INFO:teuthology.orchestra.run.smithi153.stdout:(14/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 464 kB/s | 31 kB 00:00 2024-03-28T01:58:09.010 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T01:58:09.098 INFO:teuthology.orchestra.run.smithi153.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-28T01:58:09.152 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T01:58:09.173 INFO:teuthology.orchestra.run.smithi153.stdout:(16/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 3.6 MB/s | 746 kB 00:00 2024-03-28T01:58:09.199 INFO:teuthology.orchestra.run.smithi153.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.2 MB/s | 127 kB 00:00 2024-03-28T01:58:09.206 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T01:58:09.324 INFO:teuthology.orchestra.run.smithi153.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.3 MB/s | 501 kB 00:00 2024-03-28T01:58:09.408 INFO:teuthology.orchestra.run.smithi153.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 541 kB/s | 45 kB 00:00 2024-03-28T01:58:09.491 INFO:teuthology.orchestra.run.smithi153.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.5 MB/s | 131 kB 00:00 2024-03-28T01:58:09.583 INFO:teuthology.orchestra.run.smithi153.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 162 kB 00:00 2024-03-28T01:58:09.675 INFO:teuthology.orchestra.run.smithi153.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.4 MB/s | 321 kB 00:00 2024-03-28T01:58:09.721 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T01:58:09.783 INFO:teuthology.orchestra.run.smithi153.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.7 MB/s | 298 kB 00:00 2024-03-28T01:58:09.801 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T01:58:09.875 INFO:teuthology.orchestra.run.smithi153.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.1 MB/s | 99 kB 00:00 2024-03-28T01:58:09.891 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T01:58:09.929 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T01:58:09.940 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T01:58:09.959 INFO:teuthology.orchestra.run.smithi153.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-28T01:58:09.969 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T01:58:10.018 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T01:58:10.056 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T01:58:10.086 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T01:58:10.301 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T01:58:10.508 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T01:58:10.518 INFO:teuthology.orchestra.run.smithi153.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 5.5 MB/s | 3.1 MB 00:00 2024-03-28T01:58:10.584 INFO:teuthology.orchestra.run.smithi153.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-28T01:58:10.596 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T01:58:10.640 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T01:58:10.660 INFO:teuthology.orchestra.run.smithi153.stdout:(28/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 3.5 MB/s | 5.1 MB 00:01 2024-03-28T01:58:10.685 INFO:teuthology.orchestra.run.smithi106.stdout:(10/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 2.7 MB/s | 20 MB 00:07 2024-03-28T01:58:10.685 INFO:teuthology.orchestra.run.smithi153.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2145.g 338 kB/s | 34 kB 00:00 2024-03-28T01:58:10.719 INFO:teuthology.orchestra.run.smithi153.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.3 MB/s | 139 kB 00:00 2024-03-28T01:58:10.784 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T01:58:10.844 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T01:58:10.861 INFO:teuthology.orchestra.run.smithi106.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.2 MB/s | 220 kB 00:00 2024-03-28T01:58:10.899 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T01:58:10.952 INFO:teuthology.orchestra.run.smithi106.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 295 kB/s | 27 kB 00:00 2024-03-28T01:58:10.972 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T01:58:11.011 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T01:58:11.028 INFO:teuthology.orchestra.run.smithi106.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 416 kB/s | 31 kB 00:00 2024-03-28T01:58:11.047 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T01:58:11.094 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T01:58:11.130 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T01:58:11.173 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T01:58:11.215 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T01:58:11.328 INFO:teuthology.orchestra.run.smithi153.stdout:(31/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 9.7 MB/s | 48 MB 00:04 2024-03-28T01:58:11.345 INFO:teuthology.orchestra.run.smithi106.stdout:(14/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 2.7 MB/s | 17 MB 00:06 2024-03-28T01:58:11.369 INFO:teuthology.orchestra.run.smithi153.stdout:(32/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 2.6 MB/s | 1.8 MB 00:00 2024-03-28T01:58:11.370 INFO:teuthology.orchestra.run.smithi106.stdout:(15/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 2.1 MB/s | 746 kB 00:00 2024-03-28T01:58:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T01:58:11.403 INFO:teuthology.orchestra.run.smithi153.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.2 MB/s | 248 kB 00:00 2024-03-28T01:58:11.420 INFO:teuthology.orchestra.run.smithi106.stdout:(16/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.1 MB/s | 163 kB 00:00 2024-03-28T01:58:11.428 INFO:teuthology.orchestra.run.smithi153.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 844 kB/s | 49 kB 00:00 2024-03-28T01:58:11.445 INFO:teuthology.orchestra.run.smithi153.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 394 kB/s | 16 kB 00:00 2024-03-28T01:58:11.445 INFO:teuthology.orchestra.run.smithi106.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.6 MB/s | 127 kB 00:00 2024-03-28T01:58:11.587 INFO:teuthology.orchestra.run.smithi153.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.6 MB/s | 264 kB 00:00 2024-03-28T01:58:11.637 INFO:teuthology.orchestra.run.smithi153.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.8 MB/s | 748 kB 00:00 2024-03-28T01:58:11.654 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T01:58:11.662 INFO:teuthology.orchestra.run.smithi106.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 2.0 MB/s | 501 kB 00:00 2024-03-28T01:58:11.671 INFO:teuthology.orchestra.run.smithi153.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 218 kB/s | 18 kB 00:00 2024-03-28T01:58:11.760 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T01:58:11.762 INFO:teuthology.orchestra.run.smithi106.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 451 kB/s | 45 kB 00:00 2024-03-28T01:58:11.801 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T01:58:11.863 INFO:teuthology.orchestra.run.smithi106.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.3 MB/s | 131 kB 00:00 2024-03-28T01:58:11.871 INFO:teuthology.orchestra.run.smithi153.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 662 kB 00:00 2024-03-28T01:58:11.903 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T01:58:11.913 INFO:teuthology.orchestra.run.smithi153.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-28T01:58:11.938 INFO:teuthology.orchestra.run.smithi106.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 162 kB 00:00 2024-03-28T01:58:11.942 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T01:58:11.963 INFO:teuthology.orchestra.run.smithi153.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T01:58:11.982 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T01:58:12.007 INFO:teuthology.orchestra.run.smithi153.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 763 kB/s | 32 kB 00:00 2024-03-28T01:58:12.043 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T01:58:12.043 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T01:58:12.048 INFO:teuthology.orchestra.run.smithi153.stdout:(43/174): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 920 kB 00:00 2024-03-28T01:58:12.063 INFO:teuthology.orchestra.run.smithi106.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.5 MB/s | 321 kB 00:00 2024-03-28T01:58:12.105 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T01:58:12.197 INFO:teuthology.orchestra.run.smithi106.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.2 MB/s | 298 kB 00:00 2024-03-28T01:58:12.215 INFO:teuthology.orchestra.run.smithi153.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-03-28T01:58:12.240 INFO:teuthology.orchestra.run.smithi153.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 90 kB/s | 17 kB 00:00 2024-03-28T01:58:12.241 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T01:58:12.272 INFO:teuthology.orchestra.run.smithi106.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-28T01:58:12.282 INFO:teuthology.orchestra.run.smithi153.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 242 kB/s | 10 kB 00:00 2024-03-28T01:58:12.307 INFO:teuthology.orchestra.run.smithi153.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-28T01:58:12.332 INFO:teuthology.orchestra.run.smithi153.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-28T01:58:12.347 INFO:teuthology.orchestra.run.smithi106.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-28T01:58:12.349 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T01:58:12.366 INFO:teuthology.orchestra.run.smithi153.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-28T01:58:12.400 INFO:teuthology.orchestra.run.smithi153.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-28T01:58:12.408 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T01:58:12.425 INFO:teuthology.orchestra.run.smithi153.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 806 kB/s | 46 kB 00:00 2024-03-28T01:58:12.450 INFO:teuthology.orchestra.run.smithi153.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T01:58:12.456 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T01:58:12.484 INFO:teuthology.orchestra.run.smithi153.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-28T01:58:12.509 INFO:teuthology.orchestra.run.smithi153.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-28T01:58:12.511 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T01:58:12.532 INFO:teuthology.orchestra.run.smithi153.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 715 kB/s | 34 kB 00:00 2024-03-28T01:58:12.691 INFO:teuthology.orchestra.run.smithi153.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-03-28T01:58:12.696 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T01:58:12.725 INFO:teuthology.orchestra.run.smithi153.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-28T01:58:12.746 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T01:58:12.750 INFO:teuthology.orchestra.run.smithi153.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T01:58:12.834 INFO:teuthology.orchestra.run.smithi153.stdout:(59/174): ceph-mgr-diskprediction-local-19.0.0- 3.5 MB/s | 7.4 MB 00:02 2024-03-28T01:58:12.884 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T01:58:12.927 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T01:58:12.958 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T01:58:12.968 INFO:teuthology.orchestra.run.smithi153.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 5.8 MB 00:00 2024-03-28T01:58:12.981 INFO:teuthology.orchestra.run.smithi106.stdout:(26/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 3.4 MB/s | 5.1 MB 00:01 2024-03-28T01:58:12.990 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T01:58:12.993 INFO:teuthology.orchestra.run.smithi153.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 101 kB/s | 24 kB 00:00 2024-03-28T01:58:13.017 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T01:58:13.018 INFO:teuthology.orchestra.run.smithi153.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 100 kB/s | 18 kB 00:00 2024-03-28T01:58:13.043 INFO:teuthology.orchestra.run.smithi153.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-28T01:58:13.056 INFO:teuthology.orchestra.run.smithi106.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-28T01:58:13.061 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T01:58:13.068 INFO:teuthology.orchestra.run.smithi153.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-28T01:58:13.099 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T01:58:13.131 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T01:58:13.140 INFO:teuthology.orchestra.run.smithi106.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 407 kB/s | 34 kB 00:00 2024-03-28T01:58:13.161 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T01:58:13.215 INFO:teuthology.orchestra.run.smithi106.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.8 MB/s | 139 kB 00:00 2024-03-28T01:58:13.219 INFO:teuthology.orchestra.run.smithi153.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-28T01:58:13.237 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T01:58:13.244 INFO:teuthology.orchestra.run.smithi153.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-28T01:58:13.269 INFO:teuthology.orchestra.run.smithi153.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 878 kB/s | 174 kB 00:00 2024-03-28T01:58:13.271 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T01:58:13.295 INFO:teuthology.orchestra.run.smithi153.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 924 kB/s | 69 kB 00:00 2024-03-28T01:58:13.305 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T01:58:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-28T01:58:13.324 INFO:teuthology.orchestra.run.smithi106.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 3.2 MB/s | 3.1 MB 00:00 2024-03-28T01:58:13.345 INFO:teuthology.orchestra.run.smithi153.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-28T01:58:13.346 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T01:58:13.371 INFO:teuthology.orchestra.run.smithi153.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-28T01:58:13.381 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T01:58:13.396 INFO:teuthology.orchestra.run.smithi153.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-03-28T01:58:13.413 INFO:teuthology.orchestra.run.smithi153.stdout:(73/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T01:58:13.429 INFO:teuthology.orchestra.run.smithi153.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 737 kB/s | 43 kB 00:00 2024-03-28T01:58:13.469 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T01:58:13.560 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T01:58:13.612 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T01:58:13.643 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T01:58:13.646 INFO:teuthology.orchestra.run.smithi153.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 179 kB/s | 39 kB 00:00 2024-03-28T01:58:13.690 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T01:58:13.730 INFO:teuthology.orchestra.run.smithi153.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 137 kB/s | 41 kB 00:00 2024-03-28T01:58:13.757 INFO:teuthology.orchestra.run.smithi106.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 3.3 MB/s | 1.8 MB 00:00 2024-03-28T01:58:13.787 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T01:58:13.844 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T01:58:13.891 INFO:teuthology.orchestra.run.smithi106.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 1.8 MB/s | 248 kB 00:00 2024-03-28T01:58:13.956 INFO:teuthology.orchestra.run.smithi153.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-03-28T01:58:13.958 INFO:teuthology.orchestra.run.smithi106.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 738 kB/s | 49 kB 00:00 2024-03-28T01:58:13.981 INFO:teuthology.orchestra.run.smithi153.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 898 kB/s | 300 kB 00:00 2024-03-28T01:58:14.006 INFO:teuthology.orchestra.run.smithi153.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 207 kB/s | 56 kB 00:00 2024-03-28T01:58:14.031 INFO:teuthology.orchestra.run.smithi153.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 842 kB/s | 63 kB 00:00 2024-03-28T01:58:14.056 INFO:teuthology.orchestra.run.smithi153.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-28T01:58:14.081 INFO:teuthology.orchestra.run.smithi153.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-28T01:58:14.107 INFO:teuthology.orchestra.run.smithi153.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-28T01:58:14.132 INFO:teuthology.orchestra.run.smithi153.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-28T01:58:14.166 INFO:teuthology.orchestra.run.smithi153.stdout:(85/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T01:58:14.191 INFO:teuthology.orchestra.run.smithi153.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-28T01:58:14.216 INFO:teuthology.orchestra.run.smithi153.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-28T01:58:14.275 INFO:teuthology.orchestra.run.smithi106.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 52 kB/s | 16 kB 00:00 2024-03-28T01:58:14.433 INFO:teuthology.orchestra.run.smithi106.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.6 MB/s | 264 kB 00:00 2024-03-28T01:58:14.569 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T01:58:14.608 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T01:58:14.642 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T01:58:14.690 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T01:58:14.787 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T01:58:14.808 INFO:teuthology.orchestra.run.smithi106.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 1.9 MB/s | 748 kB 00:00 2024-03-28T01:58:14.826 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T01:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 182 kB/s | 18 kB 00:00 2024-03-28T01:58:15.003 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T01:58:15.032 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T01:58:15.110 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T01:58:15.191 INFO:teuthology.orchestra.run.smithi153.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 187 kB/s | 187 kB 00:00 2024-03-28T01:58:15.251 INFO:teuthology.orchestra.run.smithi106.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 1.9 MB/s | 662 kB 00:00 2024-03-28T01:58:15.258 INFO:teuthology.orchestra.run.smithi153.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 187 kB/s | 195 kB 00:01 2024-03-28T01:58:15.318 INFO:teuthology.orchestra.run.smithi106.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-28T01:58:15.360 INFO:teuthology.orchestra.run.smithi106.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-28T01:58:15.402 INFO:teuthology.orchestra.run.smithi106.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-28T01:58:15.433 INFO:teuthology.orchestra.run.smithi153.stdout:(90/174): elfutils-libelf-devel-0.190-2.el9.x86 123 kB/s | 21 kB 00:00 2024-03-28T01:58:15.435 INFO:teuthology.orchestra.run.smithi106.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 947 kB/s | 32 kB 00:00 2024-03-28T01:58:15.550 INFO:teuthology.orchestra.run.smithi153.stdout:(91/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 280 kB/s | 25 kB 00:00 2024-03-28T01:58:15.577 INFO:teuthology.orchestra.run.smithi106.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-03-28T01:58:15.611 INFO:teuthology.orchestra.run.smithi106.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 513 kB/s | 17 kB 00:00 2024-03-28T01:58:15.636 INFO:teuthology.orchestra.run.smithi106.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 404 kB/s | 10 kB 00:00 2024-03-28T01:58:15.720 INFO:teuthology.orchestra.run.smithi106.stdout:(46/174): ceph-mgr-diskprediction-local-19.0.0- 3.1 MB/s | 7.4 MB 00:02 2024-03-28T01:58:15.745 INFO:teuthology.orchestra.run.smithi106.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-03-28T01:58:15.746 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T01:58:15.762 INFO:teuthology.orchestra.run.smithi106.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-28T01:58:15.796 INFO:teuthology.orchestra.run.smithi106.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-28T01:58:15.829 INFO:teuthology.orchestra.run.smithi106.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-28T01:58:15.855 INFO:teuthology.orchestra.run.smithi106.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 801 kB/s | 46 kB 00:00 2024-03-28T01:58:15.863 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T01:58:15.880 INFO:teuthology.orchestra.run.smithi106.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T01:58:15.913 INFO:teuthology.orchestra.run.smithi106.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-28T01:58:15.939 INFO:teuthology.orchestra.run.smithi106.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-28T01:58:15.964 INFO:teuthology.orchestra.run.smithi106.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2024-03-28T01:58:15.992 INFO:teuthology.orchestra.run.smithi153.stdout:(92/174): python3-requests-2.25.1-8.el9.noarch. 156 kB/s | 125 kB 00:00 2024-03-28T01:58:16.126 INFO:teuthology.orchestra.run.smithi153.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 415 kB/s | 803 kB 00:01 2024-03-28T01:58:16.368 INFO:teuthology.orchestra.run.smithi153.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 4.2 MB/s | 1.6 MB 00:00 2024-03-28T01:58:16.424 INFO:teuthology.orchestra.run.smithi106.stdout:(56/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 5.5 MB/s | 48 MB 00:08 2024-03-28T01:58:16.443 INFO:teuthology.orchestra.run.smithi153.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-03-28T01:58:16.507 INFO:teuthology.orchestra.run.smithi106.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 8.4 MB/s | 4.7 MB 00:00 2024-03-28T01:58:16.519 INFO:teuthology.orchestra.run.smithi153.stdout:(96/174): postgresql-contrib-13.14-1.el9.x86_64 2.3 MB/s | 911 kB 00:00 2024-03-28T01:58:16.548 INFO:teuthology.orchestra.run.smithi106.stdout:(58/174): protobuf-3.14.0-13.el9.x86_64.rpm 1.7 MB/s | 1.0 MB 00:00 2024-03-28T01:58:16.574 INFO:teuthology.orchestra.run.smithi106.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 527 kB/s | 78 kB 00:00 2024-03-28T01:58:16.652 INFO:teuthology.orchestra.run.smithi153.stdout:(97/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.8 MB/s | 4.1 MB 00:01 2024-03-28T01:58:16.677 INFO:teuthology.orchestra.run.smithi153.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-28T01:58:16.757 INFO:teuthology.orchestra.run.smithi106.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-28T01:58:16.778 INFO:teuthology.orchestra.run.smithi153.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-03-28T01:58:16.782 INFO:teuthology.orchestra.run.smithi106.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 118 kB/s | 24 kB 00:00 2024-03-28T01:58:16.808 INFO:teuthology.orchestra.run.smithi106.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 78 kB/s | 18 kB 00:00 2024-03-28T01:58:16.828 INFO:teuthology.orchestra.run.smithi153.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-03-28T01:58:16.833 INFO:teuthology.orchestra.run.smithi106.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-28T01:58:16.858 INFO:teuthology.orchestra.run.smithi106.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-28T01:58:16.887 INFO:teuthology.orchestra.run.smithi153.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 331 kB/s | 36 kB 00:00 2024-03-28T01:58:16.970 INFO:teuthology.orchestra.run.smithi153.stdout:(102/174): postgresql-server-13.14-1.el9.x86_64 11 MB/s | 5.8 MB 00:00 2024-03-28T01:58:16.983 INFO:teuthology.orchestra.run.smithi106.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-28T01:58:16.995 INFO:teuthology.orchestra.run.smithi153.stdout:(103/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 284 kB/s | 47 kB 00:00 2024-03-28T01:58:17.021 INFO:teuthology.orchestra.run.smithi153.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 701 kB/s | 94 kB 00:00 2024-03-28T01:58:17.050 INFO:teuthology.orchestra.run.smithi106.stdout:(66/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-28T01:58:17.077 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T01:58:17.084 INFO:teuthology.orchestra.run.smithi106.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 779 kB/s | 174 kB 00:00 2024-03-28T01:58:17.101 INFO:teuthology.orchestra.run.smithi106.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 592 kB/s | 69 kB 00:00 2024-03-28T01:58:17.126 INFO:teuthology.orchestra.run.smithi106.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 994 kB/s | 41 kB 00:00 2024-03-28T01:58:17.152 INFO:teuthology.orchestra.run.smithi106.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-28T01:58:17.157 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T01:58:17.171 INFO:teuthology.orchestra.run.smithi153.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-28T01:58:17.177 INFO:teuthology.orchestra.run.smithi106.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-28T01:58:17.188 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T01:58:17.202 INFO:teuthology.orchestra.run.smithi106.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-03-28T01:58:17.220 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T01:58:17.227 INFO:teuthology.orchestra.run.smithi106.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 863 kB/s | 43 kB 00:00 2024-03-28T01:58:17.247 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T01:58:17.252 INFO:teuthology.orchestra.run.smithi106.stdout:(74/174): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-28T01:58:17.296 INFO:teuthology.orchestra.run.smithi153.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-03-28T01:58:17.346 INFO:teuthology.orchestra.run.smithi153.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 393 kB/s | 19 kB 00:00 2024-03-28T01:58:17.397 INFO:teuthology.orchestra.run.smithi153.stdout:(108/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-03-28T01:58:17.422 INFO:teuthology.orchestra.run.smithi153.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-28T01:58:17.428 INFO:teuthology.orchestra.run.smithi106.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 193 kB/s | 39 kB 00:00 2024-03-28T01:58:17.478 INFO:teuthology.orchestra.run.smithi106.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 183 kB/s | 41 kB 00:00 2024-03-28T01:58:17.482 INFO:teuthology.orchestra.run.smithi153.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 812 kB/s | 49 kB 00:00 2024-03-28T01:58:17.516 INFO:teuthology.orchestra.run.smithi153.stdout:(111/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 218 kB/s | 26 kB 00:00 2024-03-28T01:58:17.533 INFO:teuthology.orchestra.run.smithi153.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-28T01:58:17.587 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T01:58:17.600 INFO:teuthology.orchestra.run.smithi153.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-28T01:58:17.619 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T01:58:17.683 INFO:teuthology.orchestra.run.smithi153.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 6.5 MB/s | 548 kB 00:00 2024-03-28T01:58:17.709 INFO:teuthology.orchestra.run.smithi153.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-28T01:58:17.710 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T01:58:17.734 INFO:teuthology.orchestra.run.smithi153.stdout:(116/174): luarocks-3.9.2-1.el9.noarch.rpm 694 kB/s | 151 kB 00:00 2024-03-28T01:58:17.748 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T01:58:17.753 INFO:teuthology.orchestra.run.smithi106.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-03-28T01:58:17.759 INFO:teuthology.orchestra.run.smithi153.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 576 kB/s | 43 kB 00:00 2024-03-28T01:58:17.778 INFO:teuthology.orchestra.run.smithi106.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 860 kB/s | 300 kB 00:00 2024-03-28T01:58:17.784 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T01:58:17.785 INFO:teuthology.orchestra.run.smithi153.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 427 kB/s | 32 kB 00:00 2024-03-28T01:58:17.804 INFO:teuthology.orchestra.run.smithi106.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 174 kB/s | 56 kB 00:00 2024-03-28T01:58:17.810 INFO:teuthology.orchestra.run.smithi153.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-03-28T01:58:17.829 INFO:teuthology.orchestra.run.smithi106.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-03-28T01:58:17.835 INFO:teuthology.orchestra.run.smithi153.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-28T01:58:17.854 INFO:teuthology.orchestra.run.smithi106.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-28T01:58:17.861 INFO:teuthology.orchestra.run.smithi153.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-28T01:58:17.879 INFO:teuthology.orchestra.run.smithi106.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-28T01:58:17.903 INFO:teuthology.orchestra.run.smithi153.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 527 kB/s | 35 kB 00:00 2024-03-28T01:58:17.905 INFO:teuthology.orchestra.run.smithi106.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-28T01:58:17.930 INFO:teuthology.orchestra.run.smithi106.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-28T01:58:17.936 INFO:teuthology.orchestra.run.smithi153.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-03-28T01:58:17.962 INFO:teuthology.orchestra.run.smithi153.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-28T01:58:17.972 INFO:teuthology.orchestra.run.smithi106.stdout:(85/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-28T01:58:17.997 INFO:teuthology.orchestra.run.smithi106.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-28T01:58:18.012 INFO:teuthology.orchestra.run.smithi153.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-03-28T01:58:18.022 INFO:teuthology.orchestra.run.smithi106.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-28T01:58:18.037 INFO:teuthology.orchestra.run.smithi153.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-28T01:58:18.062 INFO:teuthology.orchestra.run.smithi153.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-28T01:58:18.088 INFO:teuthology.orchestra.run.smithi153.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-03-28T01:58:18.113 INFO:teuthology.orchestra.run.smithi153.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-03-28T01:58:18.114 INFO:teuthology.orchestra.run.smithi106.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-03-28T01:58:18.138 INFO:teuthology.orchestra.run.smithi153.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-28T01:58:18.139 INFO:teuthology.orchestra.run.smithi106.stdout:(89/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-28T01:58:18.163 INFO:teuthology.orchestra.run.smithi153.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-03-28T01:58:18.190 INFO:teuthology.orchestra.run.smithi106.stdout:(90/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 803 kB 00:00 2024-03-28T01:58:18.205 INFO:teuthology.orchestra.run.smithi153.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 694 kB/s | 46 kB 00:00 2024-03-28T01:58:18.207 INFO:teuthology.orchestra.run.smithi106.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 321 kB/s | 21 kB 00:00 2024-03-28T01:58:18.231 INFO:teuthology.orchestra.run.smithi153.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-28T01:58:18.234 INFO:teuthology.orchestra.run.smithi106.stdout:(92/174): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-03-28T01:58:18.253 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T01:58:18.257 INFO:teuthology.orchestra.run.smithi106.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 379 kB/s | 25 kB 00:00 2024-03-28T01:58:18.273 INFO:teuthology.orchestra.run.smithi153.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-03-28T01:58:18.298 INFO:teuthology.orchestra.run.smithi153.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 628 kB/s | 58 kB 00:00 2024-03-28T01:58:18.300 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T01:58:18.323 INFO:teuthology.orchestra.run.smithi153.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-28T01:58:18.348 INFO:teuthology.orchestra.run.smithi153.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-03-28T01:58:18.374 INFO:teuthology.orchestra.run.smithi153.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-28T01:58:18.376 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T01:58:18.382 INFO:teuthology.orchestra.run.smithi106.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 7.1 MB/s | 911 kB 00:00 2024-03-28T01:58:18.399 INFO:teuthology.orchestra.run.smithi153.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 373 kB/s | 28 kB 00:00 2024-03-28T01:58:18.407 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T01:58:18.416 INFO:teuthology.orchestra.run.smithi106.stdout:(95/174): postgresql-13.14-1.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-28T01:58:18.424 INFO:teuthology.orchestra.run.smithi153.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-28T01:58:18.450 INFO:teuthology.orchestra.run.smithi153.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-28T01:58:18.466 INFO:teuthology.orchestra.run.smithi153.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 880 kB/s | 59 kB 00:00 2024-03-28T01:58:18.491 INFO:teuthology.orchestra.run.smithi106.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-03-28T01:58:18.492 INFO:teuthology.orchestra.run.smithi153.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 532 kB/s | 36 kB 00:00 2024-03-28T01:58:18.509 INFO:teuthology.orchestra.run.smithi153.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 866 kB/s | 51 kB 00:00 2024-03-28T01:58:18.542 INFO:teuthology.orchestra.run.smithi153.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-28T01:58:18.550 INFO:teuthology.orchestra.run.smithi106.stdout:(97/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 12 MB/s | 4.1 MB 00:00 2024-03-28T01:58:18.568 INFO:teuthology.orchestra.run.smithi153.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-28T01:58:18.593 INFO:teuthology.orchestra.run.smithi153.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-28T01:58:18.610 INFO:teuthology.orchestra.run.smithi153.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 297 kB/s | 20 kB 00:00 2024-03-28T01:58:18.633 INFO:teuthology.orchestra.run.smithi106.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-28T01:58:18.659 INFO:teuthology.orchestra.run.smithi106.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-28T01:58:18.718 INFO:teuthology.orchestra.run.smithi106.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 617 kB/s | 36 kB 00:00 2024-03-28T01:58:18.743 INFO:teuthology.orchestra.run.smithi106.stdout:(101/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-03-28T01:58:18.818 INFO:teuthology.orchestra.run.smithi106.stdout:(102/174): postgresql-server-13.14-1.el9.x86_64 14 MB/s | 5.8 MB 00:00 2024-03-28T01:58:18.843 INFO:teuthology.orchestra.run.smithi106.stdout:(103/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 378 kB/s | 47 kB 00:00 2024-03-28T01:58:18.885 INFO:teuthology.orchestra.run.smithi106.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 660 kB/s | 94 kB 00:00 2024-03-28T01:58:18.927 INFO:teuthology.orchestra.run.smithi153.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 534 kB/s | 192 kB 00:00 2024-03-28T01:58:19.037 INFO:teuthology.orchestra.run.smithi153.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2024-03-28T01:58:19.052 INFO:teuthology.orchestra.run.smithi106.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 95 kB/s | 22 kB 00:00 2024-03-28T01:58:19.279 INFO:teuthology.orchestra.run.smithi153.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.3 MB/s | 321 kB 00:00 2024-03-28T01:58:19.519 INFO:teuthology.orchestra.run.smithi106.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-03-28T01:58:19.905 INFO:teuthology.orchestra.run.smithi153.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 2.6 MB/s | 3.4 MB 00:01 2024-03-28T01:58:19.919 INFO:teuthology.orchestra.run.smithi106.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 356 kB/s | 309 kB 00:00 2024-03-28T01:58:19.970 INFO:teuthology.orchestra.run.smithi106.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 43 kB/s | 19 kB 00:00 2024-03-28T01:58:20.087 INFO:teuthology.orchestra.run.smithi106.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-03-28T01:58:20.145 INFO:teuthology.orchestra.run.smithi106.stdout:(110/174): abseil-cpp-20211102.0-3.el9.x86_64.r 435 kB/s | 548 kB 00:01 2024-03-28T01:58:20.205 INFO:teuthology.orchestra.run.smithi153.stdout:(153/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 2.5 MB/s | 3.1 MB 00:01 2024-03-28T01:58:20.254 INFO:teuthology.orchestra.run.smithi106.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-03-28T01:58:20.279 INFO:teuthology.orchestra.run.smithi106.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-03-28T01:58:20.521 INFO:teuthology.orchestra.run.smithi106.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 566 kB/s | 151 kB 00:00 2024-03-28T01:58:20.664 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T01:58:20.711 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T01:58:20.830 INFO:teuthology.orchestra.run.smithi153.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 67 kB/s | 42 kB 00:00 2024-03-28T01:58:21.255 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T01:58:21.298 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T01:58:21.304 INFO:teuthology.orchestra.run.smithi106.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 700 kB/s | 548 kB 00:00 2024-03-28T01:58:21.371 INFO:teuthology.orchestra.run.smithi106.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 768 kB/s | 838 kB 00:01 2024-03-28T01:58:21.379 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T01:58:21.379 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T01:58:21.379 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T01:58:21.379 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:21.413 INFO:teuthology.orchestra.run.smithi106.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 400 kB/s | 43 kB 00:00 2024-03-28T01:58:21.415 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T01:58:21.480 INFO:teuthology.orchestra.run.smithi106.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 296 kB/s | 32 kB 00:00 2024-03-28T01:58:21.522 INFO:teuthology.orchestra.run.smithi106.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-28T01:58:21.781 INFO:teuthology.orchestra.run.smithi106.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 572 kB/s | 172 kB 00:00 2024-03-28T01:58:21.822 INFO:teuthology.orchestra.run.smithi153.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 250 kB/s | 248 kB 00:00 2024-03-28T01:58:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 606 kB/s | 303 kB 00:00 2024-03-28T01:58:22.398 INFO:teuthology.orchestra.run.smithi106.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 303 kB/s | 35 kB 00:00 2024-03-28T01:58:22.598 INFO:teuthology.orchestra.run.smithi106.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 333 kB/s | 358 kB 00:01 2024-03-28T01:58:22.605 INFO:teuthology.orchestra.run.smithi153.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 91 kB/s | 71 kB 00:00 2024-03-28T01:58:22.690 INFO:teuthology.orchestra.run.smithi106.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 694 kB/s | 203 kB 00:00 2024-03-28T01:58:22.997 INFO:teuthology.orchestra.run.smithi153.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 607 kB/s | 1.8 MB 00:03 2024-03-28T01:58:23.023 INFO:teuthology.orchestra.run.smithi106.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 434 kB/s | 144 kB 00:00 2024-03-28T01:58:23.132 INFO:teuthology.orchestra.run.smithi106.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-03-28T01:58:23.139 INFO:teuthology.orchestra.run.smithi153.stdout:(158/174): NetworkManager-1.47.2-1.el9.x86_64.r 612 kB/s | 2.3 MB 00:03 2024-03-28T01:58:23.274 INFO:teuthology.orchestra.run.smithi106.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 125 kB/s | 18 kB 00:00 2024-03-28T01:58:23.408 INFO:teuthology.orchestra.run.smithi106.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 174 kB/s | 23 kB 00:00 2024-03-28T01:58:23.439 INFO:teuthology.orchestra.run.smithi153.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 44 kB/s | 36 kB 00:00 2024-03-28T01:58:23.533 INFO:teuthology.orchestra.run.smithi106.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 156 kB/s | 19 kB 00:00 2024-03-28T01:58:23.658 INFO:teuthology.orchestra.run.smithi106.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 157 kB/s | 20 kB 00:00 2024-03-28T01:58:24.039 INFO:teuthology.orchestra.run.smithi153.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 185 kB/s | 192 kB 00:01 2024-03-28T01:58:24.123 INFO:teuthology.orchestra.run.smithi153.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 34 kB/s | 23 kB 00:00 2024-03-28T01:58:24.282 INFO:teuthology.orchestra.run.smithi153.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 224 kB/s | 256 kB 00:01 2024-03-28T01:58:24.856 INFO:teuthology.orchestra.run.smithi153.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 133 kB/s | 108 kB 00:00 2024-03-28T01:58:25.049 INFO:teuthology.orchestra.run.smithi153.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 167 kB/s | 155 kB 00:00 2024-03-28T01:58:25.115 INFO:teuthology.orchestra.run.smithi153.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 163 kB/s | 136 kB 00:00 2024-03-28T01:58:25.225 INFO:teuthology.orchestra.run.smithi106.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 667 kB/s | 1.0 MB 00:01 2024-03-28T01:58:25.342 INFO:teuthology.orchestra.run.smithi106.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 398 kB/s | 46 kB 00:00 2024-03-28T01:58:25.459 INFO:teuthology.orchestra.run.smithi106.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 675 kB/s | 79 kB 00:00 2024-03-28T01:58:25.576 INFO:teuthology.orchestra.run.smithi106.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 495 kB/s | 58 kB 00:00 2024-03-28T01:58:25.649 INFO:teuthology.orchestra.run.smithi153.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 81 kB/s | 64 kB 00:00 2024-03-28T01:58:25.732 INFO:teuthology.orchestra.run.smithi153.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 42 kB/s | 29 kB 00:00 2024-03-28T01:58:25.893 INFO:teuthology.orchestra.run.smithi106.stdout:(134/174): python3-grpcio-1.46.7-10.el9.x86_64. 633 kB/s | 2.0 MB 00:03 2024-03-28T01:58:26.002 INFO:teuthology.orchestra.run.smithi106.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 152 kB/s | 16 kB 00:00 2024-03-28T01:58:26.035 INFO:teuthology.orchestra.run.smithi106.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 593 kB/s | 272 kB 00:00 2024-03-28T01:58:26.135 INFO:teuthology.orchestra.run.smithi106.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-03-28T01:58:26.194 INFO:teuthology.orchestra.run.smithi106.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 468 kB/s | 90 kB 00:00 2024-03-28T01:58:26.244 INFO:teuthology.orchestra.run.smithi106.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-03-28T01:58:26.303 INFO:teuthology.orchestra.run.smithi106.stdout:(140/174): libarrow-9.0.0-10.el9.x86_64.rpm 709 kB/s | 4.4 MB 00:06 2024-03-28T01:58:26.353 INFO:teuthology.orchestra.run.smithi106.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-03-28T01:58:26.412 INFO:teuthology.orchestra.run.smithi106.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-03-28T01:58:26.462 INFO:teuthology.orchestra.run.smithi106.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 470 kB/s | 51 kB 00:00 2024-03-28T01:58:26.520 INFO:teuthology.orchestra.run.smithi106.stdout:(144/174): python3-routes-2.5.1-5.el9.noarch.rp 578 kB/s | 188 kB 00:00 2024-03-28T01:58:26.654 INFO:teuthology.orchestra.run.smithi106.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 467 kB/s | 90 kB 00:00 2024-03-28T01:58:26.763 INFO:teuthology.orchestra.run.smithi106.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-03-28T01:58:26.788 INFO:teuthology.orchestra.run.smithi106.stdout:(147/174): python3-webob-1.8.7-6.el9.noarch.rpm 611 kB/s | 230 kB 00:00 2024-03-28T01:58:27.038 INFO:teuthology.orchestra.run.smithi106.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 696 kB/s | 192 kB 00:00 2024-03-28T01:58:27.172 INFO:teuthology.orchestra.run.smithi106.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 656 kB/s | 427 kB 00:00 2024-03-28T01:58:27.457 INFO:teuthology.orchestra.run.smithi153.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 529 kB/s | 1.2 MB 00:02 2024-03-28T01:58:27.847 INFO:teuthology.orchestra.run.smithi106.stdout:(150/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 4.7 MB/s | 3.1 MB 00:00 2024-03-28T01:58:27.906 INFO:teuthology.orchestra.run.smithi106.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-28T01:58:28.048 INFO:teuthology.orchestra.run.smithi106.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.4 MB/s | 3.4 MB 00:01 2024-03-28T01:58:28.190 INFO:teuthology.orchestra.run.smithi106.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 8.1 MB/s | 2.3 MB 00:00 2024-03-28T01:58:28.256 INFO:teuthology.orchestra.run.smithi106.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 632 kB/s | 42 kB 00:00 2024-03-28T01:58:28.298 INFO:teuthology.orchestra.run.smithi106.stdout:(155/174): NetworkManager-libnm-1.47.2-1.el9.x8 7.3 MB/s | 1.8 MB 00:00 2024-03-28T01:58:28.357 INFO:teuthology.orchestra.run.smithi106.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-28T01:58:28.382 INFO:teuthology.orchestra.run.smithi106.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.9 MB/s | 248 kB 00:00 2024-03-28T01:58:28.424 INFO:teuthology.orchestra.run.smithi106.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 545 kB/s | 36 kB 00:00 2024-03-28T01:58:28.475 INFO:teuthology.orchestra.run.smithi106.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-28T01:58:28.533 INFO:teuthology.orchestra.run.smithi106.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-28T01:58:28.558 INFO:teuthology.orchestra.run.smithi106.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 275 kB/s | 23 kB 00:00 2024-03-28T01:58:28.617 INFO:teuthology.orchestra.run.smithi106.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-28T01:58:28.643 INFO:teuthology.orchestra.run.smithi106.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-28T01:58:28.710 INFO:teuthology.orchestra.run.smithi106.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-28T01:58:28.735 INFO:teuthology.orchestra.run.smithi106.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 696 kB/s | 64 kB 00:00 2024-03-28T01:58:28.793 INFO:teuthology.orchestra.run.smithi106.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 344 kB/s | 29 kB 00:00 2024-03-28T01:58:28.885 INFO:teuthology.orchestra.run.smithi106.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 8.1 MB/s | 1.2 MB 00:00 2024-03-28T01:58:29.008 INFO:teuthology.orchestra.run.smithi153.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 300 kB/s | 465 kB 00:01 2024-03-28T01:58:29.010 INFO:teuthology.orchestra.run.smithi106.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.9 MB/s | 2.2 MB 00:00 2024-03-28T01:58:29.044 INFO:teuthology.orchestra.run.smithi106.stdout:(169/174): thrift-0.15.0-2.el9.x86_64.rpm 718 kB/s | 1.6 MB 00:02 2024-03-28T01:58:29.086 INFO:teuthology.orchestra.run.smithi106.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 749 kB/s | 31 kB 00:00 2024-03-28T01:58:29.128 INFO:teuthology.orchestra.run.smithi106.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 3.9 MB/s | 465 kB 00:00 2024-03-28T01:58:29.187 INFO:teuthology.orchestra.run.smithi106.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 74 kB 00:00 2024-03-28T01:58:29.228 INFO:teuthology.orchestra.run.smithi106.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.7 MB/s | 2.3 MB 00:00 2024-03-28T01:58:29.258 INFO:teuthology.orchestra.run.smithi153.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 612 kB/s | 2.2 MB 00:03 2024-03-28T01:58:29.278 INFO:teuthology.orchestra.run.smithi106.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.6 MB/s | 1.1 MB 00:00 2024-03-28T01:58:29.286 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-03-28T01:58:29.286 INFO:teuthology.orchestra.run.smithi106.stdout:Total 8.4 MB/s | 225 MB 00:26 2024-03-28T01:58:29.616 INFO:teuthology.orchestra.run.smithi153.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 602 kB/s | 2.3 MB 00:03 2024-03-28T01:58:29.641 INFO:teuthology.orchestra.run.smithi153.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 49 kB/s | 31 kB 00:00 2024-03-28T01:58:29.816 INFO:teuthology.orchestra.run.smithi153.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 374 kB/s | 74 kB 00:00 2024-03-28T01:58:29.845 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-03-28T01:58:30.120 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-03-28T01:58:30.120 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-03-28T01:58:31.400 INFO:teuthology.orchestra.run.smithi153.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 519 kB/s | 1.1 MB 00:02 2024-03-28T01:58:31.406 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-28T01:58:31.406 INFO:teuthology.orchestra.run.smithi153.stdout:Total 7.7 MB/s | 225 MB 00:29 2024-03-28T01:58:31.973 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-28T01:58:32.136 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-03-28T01:58:32.137 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-03-28T01:58:32.251 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-28T01:58:32.251 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-28T01:58:32.878 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T01:58:32.878 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-03-28T01:58:32.878 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-03-28T01:58:32.878 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-03-28T01:58:32.878 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-03-28T01:58:32.878 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-03-28T01:58:32.879 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /root 2024-03-28T01:58:32.879 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-03-28T01:58:32.879 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:33.037 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T01:58:33.055 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T01:58:33.055 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:58:33.055 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T01:58:33.055 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:34.084 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-28T01:58:34.084 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-28T01:58:34.556 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T01:58:34.582 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T01:58:34.582 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:58:34.582 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T01:58:34.582 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T01:58:34.582 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T01:58:34.582 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:34.807 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T01:58:34.829 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T01:58:34.829 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:58:34.829 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T01:58:34.829 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T01:58:34.830 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T01:58:34.830 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:35.276 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T01:58:35.299 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T01:58:35.299 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:58:35.299 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T01:58:35.299 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T01:58:35.299 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T01:58:35.299 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:35.394 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T01:58:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T01:58:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T01:58:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T01:58:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T01:58:35.932 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T01:58:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T01:58:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T01:58:37.180 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T01:58:37.217 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T01:58:37.318 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T01:58:37.482 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T01:58:37.511 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T01:58:37.534 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T01:58:37.534 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:58:37.534 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T01:58:37.534 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T01:58:37.534 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T01:58:37.534 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:37.578 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T01:58:37.589 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T01:58:37.634 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T01:58:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T01:58:38.555 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T01:58:38.578 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T01:58:38.578 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:58:38.578 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T01:58:38.578 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T01:58:38.578 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T01:58:38.578 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:38.625 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T01:58:38.644 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T01:58:38.644 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:58:38.645 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T01:58:38.645 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:38.698 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T01:58:38.720 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T01:58:38.720 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:58:38.720 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T01:58:38.721 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T01:58:38.721 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T01:58:38.721 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:39.041 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T01:58:39.064 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T01:58:39.064 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:58:39.064 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T01:58:39.064 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T01:58:39.064 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T01:58:39.064 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:39.104 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-03-28T01:58:39.213 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T01:58:39.248 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T01:58:39.286 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T01:58:39.296 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T01:58:39.344 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T01:58:39.398 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T01:58:39.431 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T01:58:39.456 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T01:58:39.510 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T01:58:39.834 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T01:58:39.891 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T01:58:39.995 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T01:58:40.025 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T01:58:40.116 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T01:58:40.140 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T01:58:40.188 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T01:58:40.387 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T01:58:40.451 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T01:58:40.472 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T01:58:40.585 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T01:58:40.607 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T01:58:40.656 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T01:58:40.694 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T01:58:40.726 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T01:58:40.741 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T01:58:40.943 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T01:58:40.971 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-28T01:58:40.974 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T01:58:41.016 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T01:58:41.078 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T01:58:41.121 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T01:58:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T01:58:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T01:58:41.197 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T01:58:41.201 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T01:58:41.254 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T01:58:41.288 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T01:58:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T01:58:41.344 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T01:58:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T01:58:41.395 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T01:58:41.492 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T01:58:41.537 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T01:58:41.685 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T01:58:41.731 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T01:58:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T01:58:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T01:58:41.854 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T01:58:41.882 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T01:58:41.955 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T01:58:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T01:58:41.984 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T01:58:42.003 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T01:58:42.011 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T01:58:42.041 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T01:58:42.055 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T01:58:42.129 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T01:58:42.191 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T01:58:42.228 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T01:58:42.229 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T01:58:42.284 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T01:58:42.293 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T01:58:42.312 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T01:58:42.328 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T01:58:42.371 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T01:58:42.405 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T01:58:42.416 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T01:58:42.432 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T01:58:42.438 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T01:58:42.501 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T01:58:42.502 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T01:58:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T01:58:42.537 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T01:58:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T01:58:42.572 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T01:58:42.586 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T01:58:42.609 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T01:58:42.678 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T01:58:42.724 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T01:58:42.764 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T01:58:42.795 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T01:58:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T01:58:42.830 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T01:58:42.931 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T01:58:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T01:58:43.085 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T01:58:43.271 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T01:58:43.315 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T01:58:43.389 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T01:58:43.466 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T01:58:43.503 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T01:58:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T01:58:43.595 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T01:58:43.636 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T01:58:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T01:58:43.666 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T01:58:43.679 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T01:58:43.693 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T01:58:43.699 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T01:58:43.709 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T01:58:43.748 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T01:58:43.789 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T01:58:43.789 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T01:58:43.834 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T01:58:43.872 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T01:58:43.873 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T01:58:43.882 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T01:58:43.921 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T01:58:43.948 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T01:58:43.986 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T01:58:43.991 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T01:58:44.032 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T01:58:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T01:58:44.072 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T01:58:44.110 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T01:58:44.116 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T01:58:44.146 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T01:58:44.149 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T01:58:44.159 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T01:58:44.192 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T01:58:44.212 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T01:58:44.224 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T01:58:44.254 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T01:58:44.292 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T01:58:44.319 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T01:58:44.323 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T01:58:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T01:58:44.352 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T01:58:44.378 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T01:58:44.391 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T01:58:44.411 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T01:58:44.412 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T01:58:44.435 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T01:58:44.449 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T01:58:44.483 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T01:58:44.521 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T01:58:44.578 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T01:58:44.591 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T01:58:44.621 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T01:58:44.635 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T01:58:44.676 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T01:58:44.708 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T01:58:44.749 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T01:58:44.780 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T01:58:44.817 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T01:58:44.840 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T01:58:44.863 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T01:58:44.887 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T01:58:44.899 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T01:58:44.942 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T01:58:44.984 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T01:58:44.994 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T01:58:45.034 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T01:58:45.053 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T01:58:45.118 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T01:58:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T01:58:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T01:58:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T01:58:45.323 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T01:58:45.362 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T01:58:45.402 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T01:58:45.405 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T01:58:45.428 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T01:58:45.452 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T01:58:45.482 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T01:58:45.538 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T01:58:45.553 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T01:58:45.603 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T01:58:45.612 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T01:58:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T01:58:45.661 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T01:58:45.683 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T01:58:45.721 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T01:58:45.751 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T01:58:45.759 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T01:58:45.802 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T01:58:45.859 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T01:58:45.864 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T01:58:45.914 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T01:58:45.953 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T01:58:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T01:58:45.995 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T01:58:46.074 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T01:58:46.098 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T01:58:46.107 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T01:58:46.132 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T01:58:46.139 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T01:58:46.143 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T01:58:46.157 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T01:58:46.184 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T01:58:46.184 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T01:58:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T01:58:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T01:58:46.271 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T01:58:46.275 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T01:58:46.301 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T01:58:46.301 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T01:58:46.313 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T01:58:46.344 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T01:58:46.375 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T01:58:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T01:58:46.394 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T01:58:46.403 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T01:58:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T01:58:46.420 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T01:58:46.437 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T01:58:46.454 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T01:58:46.470 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T01:58:46.476 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T01:58:46.487 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T01:58:46.508 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T01:58:46.512 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T01:58:46.535 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T01:58:46.538 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T01:58:46.545 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T01:58:46.575 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T01:58:46.585 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T01:58:46.612 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T01:58:46.614 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T01:58:46.617 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T01:58:46.642 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T01:58:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T01:58:46.658 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T01:58:46.667 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T01:58:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T01:58:46.692 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T01:58:46.694 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T01:58:46.718 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T01:58:46.725 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T01:58:46.725 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T01:58:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T01:58:46.737 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T01:58:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T01:58:46.787 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T01:58:46.855 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T01:58:46.904 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T01:58:46.904 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T01:58:46.921 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T01:58:46.954 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T01:58:46.967 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T01:58:46.987 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T01:58:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T01:58:47.005 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T01:58:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T01:58:47.039 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T01:58:47.044 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T01:58:47.044 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T01:58:47.063 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T01:58:47.092 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T01:58:47.096 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T01:58:47.132 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T01:58:47.161 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T01:58:47.164 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T01:58:47.190 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T01:58:47.204 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T01:58:47.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T01:58:47.255 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T01:58:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T01:58:47.335 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T01:58:47.463 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T01:58:47.571 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T01:58:47.612 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T01:58:47.660 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T01:58:47.716 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T01:58:47.899 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T01:58:47.950 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T01:58:47.983 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T01:58:48.043 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T01:58:48.076 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T01:58:48.103 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T01:58:48.124 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T01:58:48.139 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T01:58:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T01:58:48.203 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T01:58:48.203 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T01:58:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T01:58:48.235 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T01:58:48.273 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T01:58:48.312 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T01:58:48.344 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T01:58:48.374 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T01:58:48.413 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T01:58:48.449 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T01:58:48.449 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T01:58:48.502 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T01:58:48.527 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T01:58:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T01:58:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T01:58:48.610 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T01:58:48.699 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T01:58:48.780 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T01:58:48.824 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T01:58:48.864 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T01:58:48.911 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T01:58:49.008 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T01:58:49.057 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T01:58:49.164 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T01:58:49.282 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T01:58:49.780 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T01:58:49.829 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T01:58:49.863 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T01:58:49.910 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T01:58:49.991 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T01:58:50.022 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T01:58:50.199 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T01:58:50.220 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T01:58:50.297 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T01:58:50.501 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T01:58:50.583 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T01:58:50.614 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T01:58:50.645 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T01:58:50.673 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T01:58:50.936 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T01:58:50.986 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T01:58:51.019 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T01:58:51.058 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T01:58:51.092 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T01:58:51.124 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T01:58:51.160 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T01:58:51.622 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T01:58:51.772 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T01:58:51.843 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T01:58:51.865 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T01:58:52.227 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T01:58:52.227 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T01:58:52.227 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T01:58:52.227 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T01:58:52.227 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T01:58:52.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T01:58:52.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T01:58:52.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T01:58:52.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T01:58:52.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T01:58:52.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T01:58:52.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T01:58:52.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T01:58:52.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T01:58:52.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T01:58:52.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T01:58:52.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T01:58:52.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T01:58:52.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T01:58:52.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T01:58:52.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T01:58:52.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T01:58:52.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T01:58:52.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T01:58:52.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T01:58:52.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T01:58:52.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T01:58:52.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T01:58:52.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T01:58:52.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T01:58:52.237 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T01:58:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T01:58:52.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T01:58:52.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T01:58:52.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T01:58:52.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T01:58:52.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T01:58:52.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T01:58:52.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T01:58:52.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T01:58:52.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T01:58:52.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T01:58:52.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T01:58:52.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T01:58:52.243 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T01:58:52.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T01:58:52.278 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T01:58:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T01:58:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T01:58:52.432 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T01:58:52.459 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T01:58:52.773 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T01:58:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T01:58:52.888 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T01:58:52.919 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T01:58:52.957 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T01:58:53.419 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T01:58:53.442 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T01:58:53.513 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T01:58:53.535 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T01:58:54.079 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T01:58:54.121 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T01:58:54.663 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T01:58:54.698 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T01:58:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T01:58:54.789 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T01:58:54.789 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T01:58:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:58:54.833 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T01:58:55.240 INFO:teuthology.orchestra.run.smithi178.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T01:58:55.241 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:58:55.243 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: jq-1.6-16.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.244 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T01:58:55.245 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T01:58:55.246 INFO:teuthology.orchestra.run.smithi178.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T01:58:55.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T01:58:55.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T01:58:55.249 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T01:58:55.250 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.251 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:58:55.252 INFO:teuthology.orchestra.run.smithi178.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T01:58:55.252 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T01:58:55.252 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T01:58:55.252 INFO:teuthology.orchestra.run.smithi178.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T01:58:55.252 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T01:58:55.252 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T01:58:55.252 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T01:58:55.252 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T01:58:55.252 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-28T01:58:55.708 DEBUG:teuthology.parallel:result is None 2024-03-28T01:58:55.760 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T01:58:55.790 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T01:58:56.325 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T01:58:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T01:58:56.450 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T01:58:56.450 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T01:58:56.450 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T01:58:56.450 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:58:56.477 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T01:59:06.441 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T01:59:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /sys 2024-03-28T01:59:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /proc 2024-03-28T01:59:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /mnt 2024-03-28T01:59:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /var/tmp 2024-03-28T01:59:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /home 2024-03-28T01:59:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /root 2024-03-28T01:59:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /tmp 2024-03-28T01:59:06.441 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:06.595 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T01:59:06.611 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T01:59:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T01:59:06.611 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:08.090 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T01:59:08.115 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T01:59:08.115 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:08.115 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T01:59:08.115 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T01:59:08.115 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T01:59:08.116 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T01:59:08.360 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T01:59:08.360 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-03-28T01:59:08.360 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-03-28T01:59:08.360 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-03-28T01:59:08.360 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-03-28T01:59:08.361 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-03-28T01:59:08.361 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-03-28T01:59:08.361 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-03-28T01:59:08.361 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:08.362 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T01:59:08.362 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:08.362 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T01:59:08.362 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T01:59:08.363 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T01:59:08.363 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:08.531 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T01:59:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T01:59:08.546 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:08.546 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T01:59:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:08.799 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T01:59:08.822 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T01:59:08.822 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:08.822 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T01:59:08.822 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T01:59:08.822 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T01:59:08.822 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:08.909 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T01:59:09.319 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T01:59:09.350 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T01:59:09.382 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T01:59:09.441 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T01:59:09.464 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T01:59:09.699 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T01:59:09.732 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T01:59:10.009 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T01:59:10.034 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T01:59:10.034 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:10.034 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T01:59:10.034 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T01:59:10.034 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T01:59:10.034 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:10.251 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T01:59:10.273 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T01:59:10.273 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:10.273 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T01:59:10.274 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T01:59:10.274 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T01:59:10.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:10.692 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T01:59:10.715 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T01:59:10.718 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T01:59:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T01:59:10.741 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:10.741 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T01:59:10.741 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T01:59:10.741 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T01:59:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:10.816 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T01:59:10.839 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T01:59:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T01:59:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T01:59:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T01:59:11.041 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:11.041 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T01:59:11.041 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T01:59:11.041 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T01:59:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:11.085 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T01:59:11.096 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T01:59:11.143 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T01:59:11.248 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T01:59:11.279 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T01:59:11.311 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T01:59:11.371 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T01:59:11.401 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T01:59:11.638 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T01:59:11.669 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T01:59:12.033 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T01:59:12.062 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T01:59:12.084 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T01:59:12.084 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:12.084 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T01:59:12.085 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T01:59:12.085 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T01:59:12.085 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:12.129 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T01:59:12.151 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T01:59:12.151 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:12.151 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T01:59:12.151 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:12.196 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T01:59:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T01:59:12.219 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:12.219 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T01:59:12.219 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T01:59:12.219 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T01:59:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:12.522 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T01:59:12.545 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T01:59:12.545 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:12.545 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T01:59:12.545 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T01:59:12.545 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T01:59:12.545 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T01:59:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T01:59:12.770 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T01:59:12.933 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T01:59:12.963 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T01:59:12.987 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T01:59:12.987 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:12.987 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T01:59:12.987 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T01:59:12.987 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T01:59:12.987 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:13.039 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T01:59:13.050 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T01:59:13.095 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T01:59:13.972 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T01:59:13.999 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T01:59:14.022 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T01:59:14.022 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:14.022 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T01:59:14.022 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T01:59:14.022 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T01:59:14.022 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:14.066 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T01:59:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T01:59:14.091 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:14.092 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T01:59:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:14.309 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T01:59:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T01:59:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T01:59:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T01:59:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T01:59:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:14.643 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T01:59:14.665 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T01:59:14.665 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T01:59:14.665 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T01:59:14.665 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T01:59:14.665 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T01:59:14.665 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:17.027 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T01:59:17.069 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T01:59:17.098 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T01:59:17.126 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T01:59:17.307 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T01:59:17.425 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T01:59:17.481 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T01:59:17.537 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T01:59:17.590 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T01:59:17.625 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T01:59:17.657 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T01:59:17.690 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T01:59:17.719 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T01:59:17.791 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T01:59:17.863 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T01:59:17.903 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T01:59:18.502 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T01:59:18.550 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T01:59:18.570 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T01:59:19.130 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T01:59:19.171 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T01:59:19.206 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T01:59:19.237 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T01:59:19.379 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T01:59:19.421 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T01:59:19.454 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T01:59:19.495 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T01:59:19.548 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T01:59:19.585 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T01:59:19.586 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T01:59:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T01:59:19.624 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T01:59:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T01:59:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T01:59:19.677 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T01:59:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T01:59:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T01:59:19.720 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T01:59:19.752 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T01:59:19.788 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T01:59:19.793 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T01:59:19.818 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T01:59:19.818 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T01:59:19.823 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T01:59:19.877 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T01:59:19.903 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T01:59:19.921 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T01:59:19.929 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T01:59:19.946 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T01:59:19.962 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T01:59:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T01:59:20.021 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T01:59:20.057 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T01:59:20.095 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T01:59:20.125 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T01:59:20.151 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T01:59:20.184 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T01:59:20.210 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T01:59:20.235 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T01:59:20.242 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T01:59:20.242 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T01:59:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T01:59:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T01:59:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T01:59:20.537 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T01:59:20.569 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T01:59:20.587 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T01:59:20.597 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T01:59:20.627 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T01:59:20.630 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T01:59:20.646 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T01:59:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T01:59:21.473 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T01:59:21.555 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T01:59:21.596 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T01:59:21.678 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T01:59:21.709 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T01:59:21.734 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T01:59:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T01:59:21.785 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T01:59:21.785 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T01:59:21.857 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T01:59:21.886 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T01:59:21.886 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T01:59:21.954 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T01:59:21.980 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T01:59:22.013 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T01:59:22.039 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T01:59:22.064 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T01:59:22.097 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T01:59:22.136 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T01:59:22.168 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T01:59:22.194 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T01:59:22.219 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T01:59:22.236 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T01:59:22.261 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T01:59:22.268 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T01:59:22.268 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T01:59:22.447 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T01:59:22.448 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T01:59:22.539 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T01:59:22.563 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T01:59:22.596 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T01:59:22.623 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T01:59:22.648 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T01:59:22.712 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T01:59:25.641 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T01:59:25.641 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T01:59:25.641 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T01:59:25.641 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T01:59:25.641 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T01:59:25.642 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T01:59:25.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T01:59:25.644 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T01:59:25.644 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T01:59:25.644 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T01:59:25.658 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T01:59:25.658 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T01:59:25.658 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T01:59:25.658 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T01:59:25.658 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T01:59:25.658 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T01:59:25.659 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T01:59:25.660 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T01:59:25.661 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T01:59:25.662 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T01:59:25.663 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T01:59:25.664 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T01:59:25.665 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T01:59:25.666 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T01:59:25.667 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T01:59:25.668 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T01:59:25.669 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T01:59:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T01:59:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T01:59:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T01:59:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T01:59:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T01:59:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T01:59:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T01:59:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T01:59:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T01:59:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T01:59:27.479 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T01:59:27.479 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T01:59:27.479 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T01:59:27.480 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T01:59:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T01:59:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T01:59:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T01:59:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T01:59:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T01:59:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T01:59:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T01:59:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T01:59:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T01:59:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T01:59:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T01:59:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T01:59:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T01:59:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T01:59:28.733 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T01:59:28.734 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:28.734 INFO:teuthology.orchestra.run.smithi106.stdout:Upgraded: 2024-03-28T01:59:28.734 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T01:59:28.734 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T01:59:28.734 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T01:59:28.734 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T01:59:28.734 INFO:teuthology.orchestra.run.smithi106.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T01:59:28.734 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T01:59:28.735 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.736 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.737 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T01:59:28.738 INFO:teuthology.orchestra.run.smithi106.stdout: jq-1.6-16.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.739 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T01:59:28.740 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.741 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T01:59:28.742 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T01:59:28.743 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T01:59:28.744 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T01:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T01:59:28.746 INFO:teuthology.orchestra.run.smithi106.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T01:59:28.747 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T01:59:28.747 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T01:59:28.747 INFO:teuthology.orchestra.run.smithi106.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T01:59:28.747 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T01:59:28.747 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-03-28T01:59:29.259 DEBUG:teuthology.parallel:result is None 2024-03-28T01:59:30.481 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T01:59:30.481 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:30.481 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-03-28T01:59:30.481 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T01:59:30.481 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T01:59:30.481 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T01:59:30.481 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T01:59:30.481 INFO:teuthology.orchestra.run.smithi153.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T01:59:30.482 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.483 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T01:59:30.485 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T01:59:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T01:59:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.488 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T01:59:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T01:59:30.490 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T01:59:30.491 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T01:59:30.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T01:59:30.493 INFO:teuthology.orchestra.run.smithi153.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T01:59:30.494 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T01:59:30.494 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T01:59:30.494 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T01:59:30.494 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-28T01:59:30.494 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-28T01:59:30.976 DEBUG:teuthology.parallel:result is None 2024-03-28T01:59:30.977 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:59:31.347 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T01:59:31.372 INFO:teuthology.orchestra.run.smithi106.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T01:59:31.373 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T01:59:31.373 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T01:59:31.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:59:31.511 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T01:59:31.533 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T01:59:31.533 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T01:59:31.534 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T01:59:31.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:59:31.833 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T01:59:31.858 INFO:teuthology.orchestra.run.smithi178.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T01:59:31.858 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T01:59:31.858 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T01:59:31.860 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T01:59:31.860 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T01:59:31.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T01:59:31.895 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T01:59:31.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T01:59:31.926 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T01:59:31.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T01:59:31.960 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T01:59:31.960 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T01:59:31.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T01:59:31.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T01:59:32.061 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T01:59:32.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T01:59:32.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T01:59:32.158 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T01:59:32.158 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T01:59:32.191 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T01:59:32.261 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T01:59:32.262 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T01:59:32.262 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T01:59:32.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T01:59:32.360 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T01:59:32.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T01:59:32.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T01:59:32.456 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T01:59:32.456 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T01:59:32.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T01:59:32.550 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T01:59:32.550 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T01:59:32.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T01:59:32.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T01:59:32.650 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T01:59:32.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T01:59:32.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T01:59:32.746 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T01:59:32.746 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T01:59:32.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T01:59:32.840 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T01:59:32.926 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2}} 2024-03-28T01:59:32.927 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:59:32.927 INFO:tasks.cephadm:Cluster fsid is d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T01:59:32.927 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T01:59:32.927 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.106', 'mon.b': '172.21.15.153', 'mon.c': '172.21.15.178'} 2024-03-28T01:59:32.928 INFO:tasks.cephadm:First mon is mon.a on smithi106 2024-03-28T01:59:32.928 INFO:tasks.cephadm:First mgr is x 2024-03-28T01:59:32.928 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T01:59:32.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo hostname $(hostname -s) 2024-03-28T01:59:32.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-03-28T01:59:32.987 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname $(hostname -s) 2024-03-28T01:59:33.013 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T01:59:33.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:59:33.149 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T01:59:33.412 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T01:59:33.413 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-28T01:59:33.414 DEBUG:teuthology.orchestra.run.smithi106:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T01:59:34.559 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 01:59 /home/ubuntu/cephtest/cephadm 2024-03-28T01:59:34.559 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T01:59:35.707 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 01:59 /home/ubuntu/cephtest/cephadm 2024-03-28T01:59:35.707 DEBUG:teuthology.orchestra.run.smithi178:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T01:59:36.769 INFO:teuthology.orchestra.run.smithi178.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 01:59 /home/ubuntu/cephtest/cephadm 2024-03-28T01:59:36.770 DEBUG:teuthology.orchestra.run.smithi106:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T01:59:36.788 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T01:59:36.807 DEBUG:teuthology.orchestra.run.smithi178:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T01:59:36.840 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-28T01:59:36.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T01:59:36.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T01:59:36.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T01:59:37.666 INFO:teuthology.orchestra.run.smithi178.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T01:59:37.683 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T01:59:37.788 INFO:teuthology.orchestra.run.smithi106.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T02:00:13.098 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:00:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T02:00:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T02:00:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "repo_digests": [ 2024-03-28T02:00:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T02:00:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:00:13.098 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:00:31.164 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-28T02:00:31.164 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T02:00:31.164 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T02:00:31.164 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-03-28T02:00:31.164 INFO:teuthology.orchestra.run.smithi153.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T02:00:31.164 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-03-28T02:00:31.164 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-28T02:00:31.184 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-28T02:00:31.184 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T02:00:31.184 INFO:teuthology.orchestra.run.smithi178.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T02:00:31.184 INFO:teuthology.orchestra.run.smithi178.stdout: "repo_digests": [ 2024-03-28T02:00:31.184 INFO:teuthology.orchestra.run.smithi178.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T02:00:31.185 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-03-28T02:00:31.185 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-28T02:00:31.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph 2024-03-28T02:00:31.243 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-03-28T02:00:31.273 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2024-03-28T02:00:31.302 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /etc/ceph 2024-03-28T02:00:31.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-03-28T02:00:31.359 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /etc/ceph 2024-03-28T02:00:31.384 INFO:tasks.cephadm:Writing seed config... 2024-03-28T02:00:31.385 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T02:00:31.385 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T02:00:31.385 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T02:00:31.386 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-28T02:00:31.387 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T02:00:31.388 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-28T02:00:31.389 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:00:31.389 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T02:00:31.406 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d222a03c-eca6-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T02:00:31.406 DEBUG:teuthology.orchestra.run.smithi106:mon.a> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T02:00:31.449 DEBUG:teuthology.orchestra.run.smithi106:mgr.x> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T02:00:31.491 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T02:00:31.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid d222a03c-eca6-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.106 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T02:00:31.690 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-03-28T02:00:31.690 INFO:teuthology.orchestra.run.smithi106.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', 'd222a03c-eca6-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.106', '--skip-admin-label'] 2024-03-28T02:00:31.716 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T02:00:31.717 INFO:teuthology.orchestra.run.smithi106.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T02:00:31.717 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying podman|docker is present... 2024-03-28T02:00:31.741 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T02:00:31.741 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying lvm2 is present... 2024-03-28T02:00:31.741 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying time synchronization is in place... 2024-03-28T02:00:31.749 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T02:00:31.749 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T02:00:31.755 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T02:00:31.755 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-28T02:00:31.762 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout enabled 2024-03-28T02:00:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout active 2024-03-28T02:00:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:Unit chronyd.service is enabled and running 2024-03-28T02:00:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:Repeating the final host check... 2024-03-28T02:00:31.797 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T02:00:31.797 INFO:teuthology.orchestra.run.smithi106.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T02:00:31.797 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl is present 2024-03-28T02:00:31.797 INFO:teuthology.orchestra.run.smithi106.stdout:lvcreate is present 2024-03-28T02:00:31.803 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T02:00:31.804 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T02:00:31.810 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T02:00:31.810 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-28T02:00:31.817 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout enabled 2024-03-28T02:00:31.823 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout active 2024-03-28T02:00:31.824 INFO:teuthology.orchestra.run.smithi106.stdout:Unit chronyd.service is enabled and running 2024-03-28T02:00:31.824 INFO:teuthology.orchestra.run.smithi106.stdout:Host looks OK 2024-03-28T02:00:31.824 INFO:teuthology.orchestra.run.smithi106.stdout:Cluster fsid: d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:00:31.824 INFO:teuthology.orchestra.run.smithi106.stdout:Acquiring lock 140157560077760 on /run/cephadm/d222a03c-eca6-11ee-95d0-87774f69a715.lock 2024-03-28T02:00:31.824 INFO:teuthology.orchestra.run.smithi106.stdout:Lock 140157560077760 acquired on /run/cephadm/d222a03c-eca6-11ee-95d0-87774f69a715.lock 2024-03-28T02:00:31.824 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying IP 172.21.15.106 port 3300 ... 2024-03-28T02:00:31.824 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying IP 172.21.15.106 port 6789 ... 2024-03-28T02:00:31.824 INFO:teuthology.orchestra.run.smithi106.stdout:Base mon IP(s) is [172.21.15.106:3300, 172.21.15.106:6789], mon addrv is [v2:172.21.15.106:3300,v1:172.21.15.106:6789] 2024-03-28T02:00:31.828 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.106 metric 100 2024-03-28T02:00:31.828 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.106 metric 100 2024-03-28T02:00:31.830 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T02:00:31.830 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-28T02:00:31.830 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-03-28T02:00:31.832 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T02:00:31.833 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T02:00:31.833 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T02:00:31.833 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-28T02:00:31.833 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce6d/64 scope link 2024-03-28T02:00:31.833 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T02:00:31.833 INFO:teuthology.orchestra.run.smithi106.stdout:Mon IP `172.21.15.106` is in CIDR network `172.21.0.0/20` 2024-03-28T02:00:31.834 INFO:teuthology.orchestra.run.smithi106.stdout:Mon IP `172.21.15.106` is in CIDR network `172.21.0.0/20` 2024-03-28T02:00:31.834 INFO:teuthology.orchestra.run.smithi106.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T02:00:31.834 INFO:teuthology.orchestra.run.smithi106.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T02:00:31.835 INFO:teuthology.orchestra.run.smithi106.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T02:00:32.306 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T02:00:32.306 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T02:00:32.306 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T02:00:32.306 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-28T02:00:32.306 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-28T02:00:32.307 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T02:00:32.307 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T02:00:33.219 INFO:teuthology.orchestra.run.smithi106.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T02:00:33.219 INFO:teuthology.orchestra.run.smithi106.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T02:00:33.219 INFO:teuthology.orchestra.run.smithi106.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T02:00:34.033 INFO:teuthology.orchestra.run.smithi106.stdout:stat: stdout 167 167 2024-03-28T02:00:34.033 INFO:teuthology.orchestra.run.smithi106.stdout:Creating initial keys... 2024-03-28T02:00:34.823 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph-authtool: stdout AQBCzwRmSImtFBAA+DA+FANWvSVHAzSdiig1GA== 2024-03-28T02:00:35.712 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph-authtool: stdout AQBDzwRm0HAyCRAAezHJ4avFYpeQ3vvi2t8luA== 2024-03-28T02:00:36.585 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph-authtool: stdout AQBEzwRmOGueARAABm+bLZjq2xjo1JfXtoNrfg== 2024-03-28T02:00:36.586 INFO:teuthology.orchestra.run.smithi106.stdout:Creating initial monmap... 2024-03-28T02:00:37.433 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T02:00:37.433 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T02:00:37.434 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:00:37.434 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T02:00:37.434 INFO:teuthology.orchestra.run.smithi106.stdout:monmaptool for a [v2:172.21.15.106:3300,v1:172.21.15.106:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T02:00:37.434 INFO:teuthology.orchestra.run.smithi106.stdout:setting min_mon_release = quincy 2024-03-28T02:00:37.434 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: set fsid to d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:00:37.434 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T02:00:37.434 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:00:37.434 INFO:teuthology.orchestra.run.smithi106.stdout:Creating mon... 2024-03-28T02:00:38.331 INFO:teuthology.orchestra.run.smithi106.stdout:create mon.a on 2024-03-28T02:00:38.533 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T02:00:38.679 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T02:00:38.857 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d222a03c-eca6-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-d222a03c-eca6-11ee-95d0-87774f69a715.target. 2024-03-28T02:00:38.857 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d222a03c-eca6-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-d222a03c-eca6-11ee-95d0-87774f69a715.target. 2024-03-28T02:00:39.337 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.a 2024-03-28T02:00:39.337 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T02:00:39.519 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d222a03c-eca6-11ee-95d0-87774f69a715.target.wants/ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-d222a03c-eca6-11ee-95d0-87774f69a715@.service. 2024-03-28T02:00:39.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:39 smithi106 podman[31572]: 2024-03-28 02:00:39.67181174 +0000 UTC m=+0.016822414 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T02:00:41.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:40 smithi106 podman[31572]: 2024-03-28 02:00:40.980022039 +0000 UTC m=+1.325032713 container create b21f57acaf7fa167ae3143e5927316099c185dd0cfbe7f4e897548be07fc5c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T02:00:43.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:42 smithi106 podman[31572]: 2024-03-28 02:00:42.72991784 +0000 UTC m=+3.074928514 container init b21f57acaf7fa167ae3143e5927316099c185dd0cfbe7f4e897548be07fc5c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:00:43.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:42 smithi106 podman[31572]: 2024-03-28 02:00:42.733996433 +0000 UTC m=+3.079007110 container start b21f57acaf7fa167ae3143e5927316099c185dd0cfbe7f4e897548be07fc5c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T02:00:43.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:42 smithi106 ceph-mon[31591]: mkfs d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:00:43.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:42 smithi106 ceph-mon[31591]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T02:00:43.535 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T02:00:43.535 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-03-28T02:00:43.542 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T02:00:43.542 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-28T02:00:43.542 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-03-28T02:00:43.542 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T02:00:43.543 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mon to start... 2024-03-28T02:00:43.543 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mon... 2024-03-28T02:00:43.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:43 smithi106 bash[31572]: b21f57acaf7fa167ae3143e5927316099c185dd0cfbe7f4e897548be07fc5c0d 2024-03-28T02:00:43.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:43 smithi106 systemd[1]: Started Ceph mon.a for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:00:44.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:43 smithi106 ceph-mon[31591]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T02:00:44.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:43 smithi106 ceph-mon[31591]: monmap e1: 1 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T02:00:44.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:43 smithi106 ceph-mon[31591]: fsmap 2024-03-28T02:00:44.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:43 smithi106 ceph-mon[31591]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T02:00:44.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:43 smithi106 ceph-mon[31591]: mgrmap e1: no daemons active 2024-03-28T02:00:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:46 smithi106 ceph-mon[31591]: from='client.? 172.21.15.106:0/3414160148' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T02:00:49.402 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout id: d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout services: 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 3s) 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout data: 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:00:49.403 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.565+0000 7fe3bca0d700 1 Processor -- start 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.565+0000 7fe3bca0d700 1 -- start start 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.566+0000 7fe3bca0d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b8106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.566+0000 7fe3bca0d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b8107400 con 0x7fe3b8106a10 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.566+0000 7fe3b659c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b8106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.566+0000 7fe3b659c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b8106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51912/0 (socket says 172.21.15.106:51912) 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.566+0000 7fe3b659c700 1 -- 172.21.15.106:0/4289717712 learned_addr learned my addr 172.21.15.106:0/4289717712 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.566+0000 7fe3b659c700 1 -- 172.21.15.106:0/4289717712 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b8107c10 con 0x7fe3b8106a10 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.566+0000 7fe3b659c700 1 --2- 172.21.15.106:0/4289717712 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b8106e30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80098a0 tx=0x7fe3a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a6a7accc008bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3b5d9b700 1 -- 172.21.15.106:0/4289717712 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3a8004160 con 0x7fe3b8106a10 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3b5d9b700 1 -- 172.21.15.106:0/4289717712 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8004300 con 0x7fe3b8106a10 2024-03-28T02:00:49.404 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3b5d9b700 1 -- 172.21.15.106:0/4289717712 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3a80036a0 con 0x7fe3b8106a10 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3bca0d700 1 -- 172.21.15.106:0/4289717712 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 msgr2=0x7fe3b8106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3bca0d700 1 --2- 172.21.15.106:0/4289717712 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b8106e30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80098a0 tx=0x7fe3a8009bb0 comp rx=0 tx=0).stop 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3bca0d700 1 -- 172.21.15.106:0/4289717712 shutdown_connections 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3bca0d700 1 --2- 172.21.15.106:0/4289717712 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b8106e30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3bca0d700 1 -- 172.21.15.106:0/4289717712 >> 172.21.15.106:0/4289717712 conn(0x7fe3b8101ff0 msgr2=0x7fe3b8104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3bca0d700 1 -- 172.21.15.106:0/4289717712 shutdown_connections 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.567+0000 7fe3bca0d700 1 -- 172.21.15.106:0/4289717712 wait complete. 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.568+0000 7fe3bca0d700 1 Processor -- start 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.568+0000 7fe3bca0d700 1 -- start start 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.568+0000 7fe3bca0d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b807ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.568+0000 7fe3bca0d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b8108140 con 0x7fe3b8106a10 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3b659c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b807ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3b659c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b807ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51922/0 (socket says 172.21.15.106:51922) 2024-03-28T02:00:49.405 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3b659c700 1 -- 172.21.15.106:0/3414160148 learned_addr learned my addr 172.21.15.106:0/3414160148 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3b659c700 1 -- 172.21.15.106:0/3414160148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b819e010 con 0x7fe3b8106a10 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3b659c700 1 --2- 172.21.15.106:0/3414160148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b807ba20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8000c00 tx=0x7fe3a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3affff700 1 -- 172.21.15.106:0/3414160148 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3a8003db0 con 0x7fe3b8106a10 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3affff700 1 -- 172.21.15.106:0/3414160148 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8011630 con 0x7fe3b8106a10 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3affff700 1 -- 172.21.15.106:0/3414160148 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3a8011950 con 0x7fe3b8106a10 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3bca0d700 1 -- 172.21.15.106:0/3414160148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b819e1e0 con 0x7fe3b8106a10 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3bca0d700 1 -- 172.21.15.106:0/3414160148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b819e600 con 0x7fe3b8106a10 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3affff700 1 -- 172.21.15.106:0/3414160148 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe3a801a460 con 0x7fe3b8106a10 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.569+0000 7fe3affff700 1 -- 172.21.15.106:0/3414160148 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe3a801b780 con 0x7fe3b8106a10 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.570+0000 7fe3bca0d700 1 -- 172.21.15.106:0/3414160148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b8069260 con 0x7fe3b8106a10 2024-03-28T02:00:49.406 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.572+0000 7fe3affff700 1 -- 172.21.15.106:0/3414160148 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe3a8031020 con 0x7fe3b8106a10 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.642+0000 7fe3bca0d700 1 -- 172.21.15.106:0/3414160148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe3b802ddb0 con 0x7fe3b8106a10 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.643+0000 7fe3affff700 1 -- 172.21.15.106:0/3414160148 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+310 (secure 0 0 0) 0x7fe3a801ba20 con 0x7fe3b8106a10 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.644+0000 7fe3bca0d700 1 -- 172.21.15.106:0/3414160148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 msgr2=0x7fe3b807ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.644+0000 7fe3bca0d700 1 --2- 172.21.15.106:0/3414160148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b807ba20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8000c00 tx=0x7fe3a8004750 comp rx=0 tx=0).stop 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.644+0000 7fe3bca0d700 1 -- 172.21.15.106:0/3414160148 shutdown_connections 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.644+0000 7fe3bca0d700 1 --2- 172.21.15.106:0/3414160148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3b8106a10 0x7fe3b807ba20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.644+0000 7fe3bca0d700 1 -- 172.21.15.106:0/3414160148 >> 172.21.15.106:0/3414160148 conn(0x7fe3b8101ff0 msgr2=0x7fe3b8190dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.644+0000 7fe3bca0d700 1 -- 172.21.15.106:0/3414160148 shutdown_connections 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:46.644+0000 7fe3bca0d700 1 -- 172.21.15.106:0/3414160148 wait complete. 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:mon is available 2024-03-28T02:00:49.407 INFO:teuthology.orchestra.run.smithi106.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T02:00:50.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:49 smithi106 ceph-mon[31591]: from='client.? 172.21.15.106:0/3754497511' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T02:00:50.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:49 smithi106 ceph-mon[31591]: from='client.? 172.21.15.106:0/3754497511' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T02:00:50.434 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:00:50.434 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T02:00:50.434 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout fsid = d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.106:3300,v1:172.21.15.106:6789] 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T02:00:50.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.829+0000 7f66670ed700 1 Processor -- start 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.829+0000 7f66670ed700 1 -- start start 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f66670ed700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f6660107f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f66670ed700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6660108510 con 0x7f6660107b20 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f6664e89700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f6660107f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f6664e89700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f6660107f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51938/0 (socket says 172.21.15.106:51938) 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f6664e89700 1 -- 172.21.15.106:0/224829629 learned_addr learned my addr 172.21.15.106:0/224829629 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f6664e89700 1 -- 172.21.15.106:0/224829629 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6660108d20 con 0x7f6660107b20 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f6664e89700 1 --2- 172.21.15.106:0/224829629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f6660107f40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f66540098a0 tx=0x7f6654009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4843e657a284020 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f665f7fe700 1 -- 172.21.15.106:0/224829629 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6654004160 con 0x7f6660107b20 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f665f7fe700 1 -- 172.21.15.106:0/224829629 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6654004300 con 0x7f6660107b20 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.830+0000 7f665f7fe700 1 -- 172.21.15.106:0/224829629 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66540036a0 con 0x7f6660107b20 2024-03-28T02:00:50.436 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.831+0000 7f66670ed700 1 -- 172.21.15.106:0/224829629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 msgr2=0x7f6660107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.831+0000 7f66670ed700 1 --2- 172.21.15.106:0/224829629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f6660107f40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f66540098a0 tx=0x7f6654009bb0 comp rx=0 tx=0).stop 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.831+0000 7f66670ed700 1 -- 172.21.15.106:0/224829629 shutdown_connections 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.831+0000 7f66670ed700 1 --2- 172.21.15.106:0/224829629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f6660107f40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.831+0000 7f66670ed700 1 -- 172.21.15.106:0/224829629 >> 172.21.15.106:0/224829629 conn(0x7f66601030e0 msgr2=0x7f6660105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.831+0000 7f66670ed700 1 -- 172.21.15.106:0/224829629 shutdown_connections 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.831+0000 7f66670ed700 1 -- 172.21.15.106:0/224829629 wait complete. 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f66670ed700 1 Processor -- start 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f66670ed700 1 -- start start 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f66670ed700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f666007db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f66670ed700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6660109250 con 0x7f6660107b20 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f6664e89700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f666007db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f6664e89700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f666007db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51954/0 (socket says 172.21.15.106:51954) 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f6664e89700 1 -- 172.21.15.106:0/3754497511 learned_addr learned my addr 172.21.15.106:0/3754497511 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:50.437 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f6664e89700 1 -- 172.21.15.106:0/3754497511 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666007e060 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f6664e89700 1 --2- 172.21.15.106:0/3754497511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f666007db20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6654000c00 tx=0x7f6654004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f665dffb700 1 -- 172.21.15.106:0/3754497511 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6654003db0 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f665dffb700 1 -- 172.21.15.106:0/3754497511 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6654011630 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f66670ed700 1 -- 172.21.15.106:0/3754497511 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666007c250 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.832+0000 7f665dffb700 1 -- 172.21.15.106:0/3754497511 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6654011950 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.833+0000 7f66670ed700 1 -- 172.21.15.106:0/3754497511 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666007c790 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.833+0000 7f665dffb700 1 -- 172.21.15.106:0/3754497511 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6654011af0 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.833+0000 7f665dffb700 1 -- 172.21.15.106:0/3754497511 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f665401c450 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.834+0000 7f66670ed700 1 -- 172.21.15.106:0/3754497511 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6660069260 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.836+0000 7f665dffb700 1 -- 172.21.15.106:0/3754497511 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6654032020 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.902+0000 7f66670ed700 1 -- 172.21.15.106:0/3754497511 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6660194820 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.905+0000 7f665dffb700 1 -- 172.21.15.106:0/3754497511 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f6660194820 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.906+0000 7f665dffb700 1 -- 172.21.15.106:0/3754497511 <== mon.0 v2:172.21.15.106:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6654004260 con 0x7f6660107b20 2024-03-28T02:00:50.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.907+0000 7f66670ed700 1 -- 172.21.15.106:0/3754497511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 msgr2=0x7f666007db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:50.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.907+0000 7f66670ed700 1 --2- 172.21.15.106:0/3754497511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f666007db20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6654000c00 tx=0x7f6654004750 comp rx=0 tx=0).stop 2024-03-28T02:00:50.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.907+0000 7f66670ed700 1 -- 172.21.15.106:0/3754497511 shutdown_connections 2024-03-28T02:00:50.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.907+0000 7f66670ed700 1 --2- 172.21.15.106:0/3754497511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660107b20 0x7f666007db20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:50.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.907+0000 7f66670ed700 1 -- 172.21.15.106:0/3754497511 >> 172.21.15.106:0/3754497511 conn(0x7f66601030e0 msgr2=0x7f6660191ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:50.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.907+0000 7f66670ed700 1 -- 172.21.15.106:0/3754497511 shutdown_connections 2024-03-28T02:00:50.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:49.907+0000 7f66670ed700 1 -- 172.21.15.106:0/3754497511 wait complete. 2024-03-28T02:00:50.439 INFO:teuthology.orchestra.run.smithi106.stdout:Generating new minimal ceph.conf... 2024-03-28T02:00:51.230 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:50 smithi106 ceph-mon[31591]: from='client.? 172.21.15.106:0/4010267426' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:00:51.497 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.874+0000 7f848bf00700 1 Processor -- start 2024-03-28T02:00:51.497 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.874+0000 7f848bf00700 1 -- start start 2024-03-28T02:00:51.497 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.874+0000 7f848bf00700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f8484106240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:51.497 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.874+0000 7f848bf00700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8484106810 con 0x7f8484105e20 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.874+0000 7f8489c9c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f8484106240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.874+0000 7f8489c9c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f8484106240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51962/0 (socket says 172.21.15.106:51962) 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.874+0000 7f8489c9c700 1 -- 172.21.15.106:0/3941525731 learned_addr learned my addr 172.21.15.106:0/3941525731 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.874+0000 7f8489c9c700 1 -- 172.21.15.106:0/3941525731 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8484108db0 con 0x7f8484105e20 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.874+0000 7f8489c9c700 1 --2- 172.21.15.106:0/3941525731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f8484106240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f84740098a0 tx=0x7f8474009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160416bcf4d1a1de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f8488c9a700 1 -- 172.21.15.106:0/3941525731 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8474004160 con 0x7f8484105e20 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f8488c9a700 1 -- 172.21.15.106:0/3941525731 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8474004300 con 0x7f8484105e20 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f8488c9a700 1 -- 172.21.15.106:0/3941525731 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8474011b50 con 0x7f8484105e20 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f848bf00700 1 -- 172.21.15.106:0/3941525731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 msgr2=0x7f8484106240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f848bf00700 1 --2- 172.21.15.106:0/3941525731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f8484106240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f84740098a0 tx=0x7f8474009bb0 comp rx=0 tx=0).stop 2024-03-28T02:00:51.498 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f848bf00700 1 -- 172.21.15.106:0/3941525731 shutdown_connections 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f848bf00700 1 --2- 172.21.15.106:0/3941525731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f8484106240 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f848bf00700 1 -- 172.21.15.106:0/3941525731 >> 172.21.15.106:0/3941525731 conn(0x7f84841017b0 msgr2=0x7f8484103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f848bf00700 1 -- 172.21.15.106:0/3941525731 shutdown_connections 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.875+0000 7f848bf00700 1 -- 172.21.15.106:0/3941525731 wait complete. 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.876+0000 7f848bf00700 1 Processor -- start 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.876+0000 7f848bf00700 1 -- start start 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.876+0000 7f848bf00700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f848419bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.876+0000 7f848bf00700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848410c160 con 0x7f8484105e20 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.876+0000 7f8489c9c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f848419bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.876+0000 7f8489c9c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f848419bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51974/0 (socket says 172.21.15.106:51974) 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.876+0000 7f8489c9c700 1 -- 172.21.15.106:0/4010267426 learned_addr learned my addr 172.21.15.106:0/4010267426 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.876+0000 7f8489c9c700 1 -- 172.21.15.106:0/4010267426 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848419c0f0 con 0x7f8484105e20 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.877+0000 7f8489c9c700 1 --2- 172.21.15.106:0/4010267426 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f848419bbb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8474009fb0 tx=0x7f8474011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:51.499 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.877+0000 7f847affd700 1 -- 172.21.15.106:0/4010267426 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f84740038b0 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.877+0000 7f847affd700 1 -- 172.21.15.106:0/4010267426 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8474011e50 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.877+0000 7f848bf00700 1 -- 172.21.15.106:0/4010267426 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848419c380 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.877+0000 7f847affd700 1 -- 172.21.15.106:0/4010267426 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8474019550 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.877+0000 7f848bf00700 1 -- 172.21.15.106:0/4010267426 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848419f070 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.877+0000 7f847affd700 1 -- 172.21.15.106:0/4010267426 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8474019b90 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.877+0000 7f847affd700 1 -- 172.21.15.106:0/4010267426 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8474024900 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.878+0000 7f848bf00700 1 -- 172.21.15.106:0/4010267426 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8484069260 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.880+0000 7f847affd700 1 -- 172.21.15.106:0/4010267426 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8474003a50 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.946+0000 7f848bf00700 1 -- 172.21.15.106:0/4010267426 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f848419f240 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.946+0000 7f847affd700 1 -- 172.21.15.106:0/4010267426 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f8474032030 con 0x7f8484105e20 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.947+0000 7f848bf00700 1 -- 172.21.15.106:0/4010267426 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 msgr2=0x7f848419bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.947+0000 7f848bf00700 1 --2- 172.21.15.106:0/4010267426 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f848419bbb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8474009fb0 tx=0x7f8474011bf0 comp rx=0 tx=0).stop 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.947+0000 7f848bf00700 1 -- 172.21.15.106:0/4010267426 shutdown_connections 2024-03-28T02:00:51.500 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.947+0000 7f848bf00700 1 --2- 172.21.15.106:0/4010267426 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8484105e20 0x7f848419bbb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:51.501 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.947+0000 7f848bf00700 1 -- 172.21.15.106:0/4010267426 >> 172.21.15.106:0/4010267426 conn(0x7f84841017b0 msgr2=0x7f84841022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:51.501 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.947+0000 7f848bf00700 1 -- 172.21.15.106:0/4010267426 shutdown_connections 2024-03-28T02:00:51.501 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:50.947+0000 7f848bf00700 1 -- 172.21.15.106:0/4010267426 wait complete. 2024-03-28T02:00:51.501 INFO:teuthology.orchestra.run.smithi106.stdout:Restarting the monitor... 2024-03-28T02:00:51.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:51 smithi106 systemd[1]: Stopping Ceph mon.a for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:00:52.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:51 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a[31587]: 2024-03-28T02:00:51.765+0000 7f15ebc6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:00:52.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:51 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a[31587]: 2024-03-28T02:00:51.765+0000 7f15ebc6b700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T02:00:52.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:51 smithi106 podman[31838]: 2024-03-28 02:00:51.939926974 +0000 UTC m=+0.283638400 container died b21f57acaf7fa167ae3143e5927316099c185dd0cfbe7f4e897548be07fc5c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:00:52.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:52 smithi106 podman[31838]: 2024-03-28 02:00:52.273794943 +0000 UTC m=+0.617506360 container cleanup b21f57acaf7fa167ae3143e5927316099c185dd0cfbe7f4e897548be07fc5c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True) 2024-03-28T02:00:52.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:52 smithi106 bash[31838]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a 2024-03-28T02:00:52.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:52 smithi106 podman[31853]: 2024-03-28 02:00:52.449780943 +0000 UTC m=+0.511410631 container remove b21f57acaf7fa167ae3143e5927316099c185dd0cfbe7f4e897548be07fc5c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:00:52.988 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:52 smithi106 systemd[1]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T02:00:52.988 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:52 smithi106 systemd[1]: Stopped Ceph mon.a for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:00:52.988 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:52 smithi106 systemd[1]: Starting Ceph mon.a for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:00:52.988 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:52 smithi106 podman[31946]: 2024-03-28 02:00:52.892728602 +0000 UTC m=+0.103530388 container create f36f9e681e52ccbc0fdafc94c356efa07f9b8be9d86ca8a88561b3a8b55da130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T02:00:52.988 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:52 smithi106 podman[31946]: 2024-03-28 02:00:52.806562553 +0000 UTC m=+0.017364304 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T02:00:53.168 INFO:teuthology.orchestra.run.smithi106.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T02:00:53.292 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 podman[31946]: 2024-03-28 02:00:53.086727475 +0000 UTC m=+0.297529224 container init f36f9e681e52ccbc0fdafc94c356efa07f9b8be9d86ca8a88561b3a8b55da130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 podman[31946]: 2024-03-28 02:00:53.09008321 +0000 UTC m=+0.300884957 container start f36f9e681e52ccbc0fdafc94c356efa07f9b8be9d86ca8a88561b3a8b55da130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: pidfile_write: ignore empty --pid-file 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: load: jerasure load: lrc 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: RocksDB version: 7.9.2 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Git sha 0 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: DB SUMMARY 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: DB Session ID: EXQRAS7L8COH2JTSZY47 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: CURRENT file: CURRENT 2024-03-28T02:00:53.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100931 ; 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.error_if_exists: 0 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.create_if_missing: 0 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.paranoid_checks: 1 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.env: 0x55e383212b20 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.info_log: 0x55e3858fb920 2024-03-28T02:00:53.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T02:00:53.296 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.statistics: (nil) 2024-03-28T02:00:53.296 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.use_fsync: 0 2024-03-28T02:00:53.296 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_log_file_size: 0 2024-03-28T02:00:53.296 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T02:00:53.296 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T02:00:53.296 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T02:00:53.296 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.allow_fallocate: 1 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.use_direct_reads: 0 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.db_log_dir: 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.wal_dir: 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T02:00:53.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.write_buffer_manager: 0x55e3858ce1e0 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.unordered_write: 0 2024-03-28T02:00:53.298 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.row_cache: None 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.wal_filter: None 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.two_write_queues: 0 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.wal_compression: 0 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.atomic_flush: 0 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T02:00:53.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.log_readahead_size: 0 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_background_jobs: 2 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_background_compactions: -1 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_subcompactions: 1 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T02:00:53.300 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_open_files: -1 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_background_flushes: -1 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Compression algorithms supported: 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T02:00:53.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: kZSTD supported: 0 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: kXpressCompression supported: 0 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: kZlibCompression supported: 1 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: kSnappyCompression supported: 1 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: kLZ4Compression supported: 1 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: kBZip2Compression supported: 0 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.merge_operator: 2024-03-28T02:00:53.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_filter: None 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e3858fba20) 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: cache_index_and_filter_blocks: 1 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: pin_top_level_index_and_filter: 1 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: index_type: 0 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: data_block_index_type: 0 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: index_shortening: 1 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: checksum: 4 2024-03-28T02:00:53.303 INFO:journalctl@ceph.mon.a.smithi106.stdout: no_block_cache: 0 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache: 0x55e385901090 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache_name: BinnedLRUCache 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache_options: 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: capacity : 536870912 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: num_shard_bits : 4 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: strict_capacity_limit : 0 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: high_pri_pool_ratio: 0.000 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache_compressed: (nil) 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: persistent_cache: (nil) 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_size: 4096 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_size_deviation: 10 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_restart_interval: 16 2024-03-28T02:00:53.304 INFO:journalctl@ceph.mon.a.smithi106.stdout: index_block_restart_interval: 1 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: metadata_block_size: 4096 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: partition_filters: 0 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: use_delta_encoding: 1 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: filter_policy: bloomfilter 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: whole_key_filtering: 1 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: verify_compression: 0 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: read_amp_bytes_per_bit: 0 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: format_version: 5 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: enable_index_compression: 1 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_align: 0 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: max_auto_readahead_size: 262144 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: prepopulate_block_cache: 0 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: initial_auto_readahead_size: 8192 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T02:00:53.305 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression: NoCompression 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.num_levels: 7 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T02:00:53.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T02:00:53.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T02:00:53.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.table_properties_collectors: 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.inplace_update_support: 0 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T02:00:53.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.bloom_locality: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.max_successive_merges: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.ttl: 2592000 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.enable_blob_files: false 2024-03-28T02:00:53.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.min_blob_size: 0 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 93fe1c2d-5743-46f5-b5f3-4a512428724f 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711591253119498, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711591253120426, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95904, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 93890, "index_size": 296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13934, "raw_average_key_size": 49, "raw_value_size": 86324, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711591253, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "93fe1c2d-5743-46f5-b5f3-4a512428724f", "db_session_id": "EXQRAS7L8COH2JTSZY47", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711591253120510, "job": 1, "event": "recovery_finished"} 2024-03-28T02:00:53.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e3859f4000 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: DB pointer 0x55e3859de000 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** DB Stats ** 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T02:00:53.312 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** Compaction Stats [default] ** 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: L0 2/0 95.49 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: Sum 2/0 95.49 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** Compaction Stats [default] ** 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-28T02:00:53.313 INFO:journalctl@ceph.mon.a.smithi106.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative compaction: 0.00 GB write, 31.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval compaction: 0.00 GB write, 31.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: Block cache BinnedLRUCache@0x55e385901090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T02:00:53.314 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: starting mon.a rank 0 at public addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] at bind addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mon.a@-1(???) e1 preinit fsid d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mon.a@-1(???).mds e1 new map 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mon.a@-1(???).mds e1 print_map 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout: e1 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout: legacy client fscid: -1 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout: No filesystems configured 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 bash[31946]: f36f9e681e52ccbc0fdafc94c356efa07f9b8be9d86ca8a88561b3a8b55da130 2024-03-28T02:00:53.315 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 systemd[1]: Started Ceph mon.a for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:00:53.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T02:00:53.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: monmap e1: 1 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T02:00:53.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: fsmap 2024-03-28T02:00:53.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T02:00:53.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:53 smithi106 ceph-mon[31965]: mgrmap e1: no daemons active 2024-03-28T02:00:55.062 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:54 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/97588463' entity='client.admin' 2024-03-28T02:00:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.625+0000 7f555060b700 1 Processor -- start 2024-03-28T02:00:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.625+0000 7f555060b700 1 -- start start 2024-03-28T02:00:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.625+0000 7f555060b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f5548107f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.625+0000 7f555060b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5548108500 con 0x7f5548107b10 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.626+0000 7f554e3a7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f5548107f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.626+0000 7f554e3a7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f5548107f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51984/0 (socket says 172.21.15.106:51984) 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.626+0000 7f554e3a7700 1 -- 172.21.15.106:0/4289291952 learned_addr learned my addr 172.21.15.106:0/4289291952 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.626+0000 7f554e3a7700 1 -- 172.21.15.106:0/4289291952 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5548108d10 con 0x7f5548107b10 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.626+0000 7f554e3a7700 1 --2- 172.21.15.106:0/4289291952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f5548107f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f55440098a0 tx=0x7f5544009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc142627ed5f825 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.626+0000 7f554d3a5700 1 -- 172.21.15.106:0/4289291952 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5544004160 con 0x7f5548107b10 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.626+0000 7f554d3a5700 1 -- 172.21.15.106:0/4289291952 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5544004300 con 0x7f5548107b10 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.626+0000 7f554d3a5700 1 -- 172.21.15.106:0/4289291952 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5544011b50 con 0x7f5548107b10 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.627+0000 7f555060b700 1 -- 172.21.15.106:0/4289291952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 msgr2=0x7f5548107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.627+0000 7f555060b700 1 --2- 172.21.15.106:0/4289291952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f5548107f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f55440098a0 tx=0x7f5544009bb0 comp rx=0 tx=0).stop 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.627+0000 7f555060b700 1 -- 172.21.15.106:0/4289291952 shutdown_connections 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.627+0000 7f555060b700 1 --2- 172.21.15.106:0/4289291952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f5548107f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.627+0000 7f555060b700 1 -- 172.21.15.106:0/4289291952 >> 172.21.15.106:0/4289291952 conn(0x7f5548103190 msgr2=0x7f55481055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.627+0000 7f555060b700 1 -- 172.21.15.106:0/4289291952 shutdown_connections 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.627+0000 7f555060b700 1 -- 172.21.15.106:0/4289291952 wait complete. 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f555060b700 1 Processor -- start 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f555060b700 1 -- start start 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f555060b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f554819bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f555060b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554810c0c0 con 0x7f5548107b10 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f554e3a7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f554819bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f554e3a7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f554819bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51986/0 (socket says 172.21.15.106:51986) 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f554e3a7700 1 -- 172.21.15.106:0/97588463 learned_addr learned my addr 172.21.15.106:0/97588463 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f554e3a7700 1 -- 172.21.15.106:0/97588463 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554819c170 con 0x7f5548107b10 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f554e3a7700 1 --2- 172.21.15.106:0/97588463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f554819bc30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5544009fb0 tx=0x7f5544011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.628+0000 7f553f7fe700 1 -- 172.21.15.106:0/97588463 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55440038b0 con 0x7f5548107b10 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.629+0000 7f553f7fe700 1 -- 172.21.15.106:0/97588463 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5544011e50 con 0x7f5548107b10 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.629+0000 7f553f7fe700 1 -- 172.21.15.106:0/97588463 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5544019550 con 0x7f5548107b10 2024-03-28T02:00:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.629+0000 7f555060b700 1 -- 172.21.15.106:0/97588463 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554819c400 con 0x7f5548107b10 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.629+0000 7f555060b700 1 -- 172.21.15.106:0/97588463 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554819cf20 con 0x7f5548107b10 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.629+0000 7f553f7fe700 1 -- 172.21.15.106:0/97588463 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5544019b90 con 0x7f5548107b10 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.629+0000 7f553f7fe700 1 -- 172.21.15.106:0/97588463 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5544024730 con 0x7f5548107b10 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.629+0000 7f555060b700 1 -- 172.21.15.106:0/97588463 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55480637c0 con 0x7f5548107b10 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.632+0000 7f553f7fe700 1 -- 172.21.15.106:0/97588463 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5544003a50 con 0x7f5548107b10 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.697+0000 7f555060b700 1 -- 172.21.15.106:0/97588463 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f554806a4c0 con 0x7f5548107b10 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.699+0000 7f553f7fe700 1 -- 172.21.15.106:0/97588463 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f55440249d0 con 0x7f5548107b10 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.700+0000 7f555060b700 1 -- 172.21.15.106:0/97588463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 msgr2=0x7f554819bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.700+0000 7f555060b700 1 --2- 172.21.15.106:0/97588463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f554819bc30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5544009fb0 tx=0x7f5544011bf0 comp rx=0 tx=0).stop 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.700+0000 7f555060b700 1 -- 172.21.15.106:0/97588463 shutdown_connections 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.700+0000 7f555060b700 1 --2- 172.21.15.106:0/97588463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5548107b10 0x7f554819bc30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.700+0000 7f555060b700 1 -- 172.21.15.106:0/97588463 >> 172.21.15.106:0/97588463 conn(0x7f5548103190 msgr2=0x7f5548103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.700+0000 7f555060b700 1 -- 172.21.15.106:0/97588463 shutdown_connections 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:53.700+0000 7f555060b700 1 -- 172.21.15.106:0/97588463 wait complete. 2024-03-28T02:00:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T02:00:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T02:00:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:Creating mgr... 2024-03-28T02:00:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T02:00:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T02:00:55.612 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.x 2024-03-28T02:00:55.612 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T02:00:55.755 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d222a03c-eca6-11ee-95d0-87774f69a715.target.wants/ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-d222a03c-eca6-11ee-95d0-87774f69a715@.service. 2024-03-28T02:00:56.084 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:55 smithi106 podman[32179]: 2024-03-28 02:00:55.996747644 +0000 UTC m=+0.120766065 container create 392dee3cdfe634b560e74602ab365baebe3a4cd95d937017103599107a3f1cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T02:00:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T02:00:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-03-28T02:00:56.346 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T02:00:56.346 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-28T02:00:56.346 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-03-28T02:00:56.346 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T02:00:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T02:00:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-03-28T02:00:56.361 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T02:00:56.361 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-28T02:00:56.361 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-03-28T02:00:56.361 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T02:00:56.361 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr to start... 2024-03-28T02:00:56.361 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr... 2024-03-28T02:00:56.485 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:56 smithi106 podman[32179]: 2024-03-28 02:00:56.223870894 +0000 UTC m=+0.347889312 container init 392dee3cdfe634b560e74602ab365baebe3a4cd95d937017103599107a3f1cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2) 2024-03-28T02:00:56.485 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:56 smithi106 podman[32179]: 2024-03-28 02:00:56.226405948 +0000 UTC m=+0.350424366 container start 392dee3cdfe634b560e74602ab365baebe3a4cd95d937017103599107a3f1cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T02:00:56.485 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:56 smithi106 bash[32179]: 392dee3cdfe634b560e74602ab365baebe3a4cd95d937017103599107a3f1cc4 2024-03-28T02:00:56.486 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:56 smithi106 systemd[1]: Started Ceph mgr.x for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:00:57.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:00:56 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2609148363' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T02:00:57.084 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:56 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:56.775+0000 7f816c22d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T02:00:57.084 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:56 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:56.873+0000 7f816c22d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T02:00:57.435 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:00:57.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-28T02:00:57.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:00:57.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T02:00:57.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T02:00:57.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T02:00:57.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T02:00:57.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:00:57.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T02:00:57.438 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T02:00:57.439 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T02:00:57.447 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:00:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T02:00:42.806958+0000", 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.818+0000 7f906a7e7700 1 Processor -- start 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.818+0000 7f906a7e7700 1 -- start start 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.818+0000 7f906a7e7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f9064106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.818+0000 7f906a7e7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9064107400 con 0x7f9064106a10 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.818+0000 7f9063fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f9064106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.819+0000 7f9063fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f9064106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52018/0 (socket says 172.21.15.106:52018) 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.819+0000 7f9063fff700 1 -- 172.21.15.106:0/2348773187 learned_addr learned my addr 172.21.15.106:0/2348773187 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:57.449 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.819+0000 7f9063fff700 1 -- 172.21.15.106:0/2348773187 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9064107c10 con 0x7f9064106a10 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.819+0000 7f9063fff700 1 --2- 172.21.15.106:0/2348773187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f9064106e30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f905800bb40 tx=0x7f905800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144c0b1f47acac14 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.819+0000 7f9062ffd700 1 -- 172.21.15.106:0/2348773187 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f905800dbe0 con 0x7f9064106a10 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.819+0000 7f9062ffd700 1 -- 172.21.15.106:0/2348773187 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90580044f0 con 0x7f9064106a10 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.819+0000 7f9062ffd700 1 -- 172.21.15.106:0/2348773187 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9058010470 con 0x7f9064106a10 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.820+0000 7f906a7e7700 1 -- 172.21.15.106:0/2348773187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 msgr2=0x7f9064106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.820+0000 7f906a7e7700 1 --2- 172.21.15.106:0/2348773187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f9064106e30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f905800bb40 tx=0x7f905800be50 comp rx=0 tx=0).stop 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.820+0000 7f906a7e7700 1 -- 172.21.15.106:0/2348773187 shutdown_connections 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.820+0000 7f906a7e7700 1 --2- 172.21.15.106:0/2348773187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f9064106e30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.820+0000 7f906a7e7700 1 -- 172.21.15.106:0/2348773187 >> 172.21.15.106:0/2348773187 conn(0x7f9064101ff0 msgr2=0x7f9064104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.820+0000 7f906a7e7700 1 -- 172.21.15.106:0/2348773187 shutdown_connections 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.820+0000 7f906a7e7700 1 -- 172.21.15.106:0/2348773187 wait complete. 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.820+0000 7f906a7e7700 1 Processor -- start 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.820+0000 7f906a7e7700 1 -- start start 2024-03-28T02:00:57.450 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.821+0000 7f906a7e7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f906419dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.821+0000 7f906a7e7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906407af80 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.821+0000 7f9063fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f906419dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.821+0000 7f9063fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f906419dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52034/0 (socket says 172.21.15.106:52034) 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.821+0000 7f9063fff700 1 -- 172.21.15.106:0/2609148363 learned_addr learned my addr 172.21.15.106:0/2609148363 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.821+0000 7f9063fff700 1 -- 172.21.15.106:0/2609148363 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906419e290 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.821+0000 7f9063fff700 1 --2- 172.21.15.106:0/2609148363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f906419dd50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f905800b340 tx=0x7f90580040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.821+0000 7f90617fa700 1 -- 172.21.15.106:0/2609148363 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9058010720 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.822+0000 7f90617fa700 1 -- 172.21.15.106:0/2609148363 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f905801d070 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.822+0000 7f906a7e7700 1 -- 172.21.15.106:0/2609148363 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f906419e520 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.822+0000 7f90617fa700 1 -- 172.21.15.106:0/2609148363 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9058021b80 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.822+0000 7f906a7e7700 1 -- 172.21.15.106:0/2609148363 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90641a1210 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.822+0000 7f90617fa700 1 -- 172.21.15.106:0/2609148363 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f905801f030 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.822+0000 7f90617fa700 1 -- 172.21.15.106:0/2609148363 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f905802ade0 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.822+0000 7f906a7e7700 1 -- 172.21.15.106:0/2609148363 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9064069260 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.825+0000 7f90617fa700 1 -- 172.21.15.106:0/2609148363 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9058015070 con 0x7f9064106a10 2024-03-28T02:00:57.451 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.897+0000 7f906a7e7700 1 -- 172.21.15.106:0/2609148363 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f906402dc70 con 0x7f9064106a10 2024-03-28T02:00:57.452 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.897+0000 7f90617fa700 1 -- 172.21.15.106:0/2609148363 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f9058004460 con 0x7f9064106a10 2024-03-28T02:00:57.452 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.898+0000 7f906a7e7700 1 -- 172.21.15.106:0/2609148363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 msgr2=0x7f906419dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:00:57.452 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.898+0000 7f906a7e7700 1 --2- 172.21.15.106:0/2609148363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f906419dd50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f905800b340 tx=0x7f90580040b0 comp rx=0 tx=0).stop 2024-03-28T02:00:57.452 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.898+0000 7f906a7e7700 1 -- 172.21.15.106:0/2609148363 shutdown_connections 2024-03-28T02:00:57.452 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.898+0000 7f906a7e7700 1 --2- 172.21.15.106:0/2609148363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9064106a10 0x7f906419dd50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:00:57.452 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.898+0000 7f906a7e7700 1 -- 172.21.15.106:0/2609148363 >> 172.21.15.106:0/2609148363 conn(0x7f9064101ff0 msgr2=0x7f90641947f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:00:57.452 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.899+0000 7f906a7e7700 1 -- 172.21.15.106:0/2609148363 shutdown_connections 2024-03-28T02:00:57.452 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:56.899+0000 7f906a7e7700 1 -- 172.21.15.106:0/2609148363 wait complete. 2024-03-28T02:00:57.452 INFO:teuthology.orchestra.run.smithi106.stdout:mgr not available, waiting (1/15)... 2024-03-28T02:00:57.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:57 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:57.528+0000 7f816c22d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T02:00:57.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:57 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:57.619+0000 7f816c22d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T02:00:57.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:57 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:57.756+0000 7f816c22d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T02:00:58.183 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:57 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:57.849+0000 7f816c22d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T02:00:58.184 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:57 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:57.930+0000 7f816c22d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T02:00:58.184 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:58 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:58.189+0000 7f816c22d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T02:00:58.467 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:58 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:58.296+0000 7f816c22d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T02:00:58.467 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:58 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:58.380+0000 7f816c22d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T02:00:58.467 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:58 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:58.472+0000 7f816c22d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T02:00:58.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:58 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:58.644+0000 7f816c22d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T02:00:59.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:59 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:59.606+0000 7f816c22d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T02:00:59.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:00:59 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:00:59.737+0000 7f816c22d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T02:01:00.156 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:00 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2007791673' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T02:01:00.156 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:00.073+0000 7f816c22d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T02:01:00.156 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:00.161+0000 7f816c22d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T02:01:00.533 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:00.356+0000 7f816c22d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:01:00.533 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:00.538+0000 7f816c22d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T02:01:00.635 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:01:00.635 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-28T02:01:00.635 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:01:00.635 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T02:01:00.635 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T02:01:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:00.637 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T02:01:00.637 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:00.637 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T02:01:00.637 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T02:01:00.637 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T02:01:00.645 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T02:01:00.646 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T02:00:42.806958+0000", 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.888+0000 7fb2308d6700 1 Processor -- start 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.888+0000 7fb2308d6700 1 -- start start 2024-03-28T02:01:00.647 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.888+0000 7fb2308d6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c10a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.888+0000 7fb2308d6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c07be20 con 0x7fb22c1085c0 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.888+0000 7fb22a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c10a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.888+0000 7fb22a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c10a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54874/0 (socket says 172.21.15.106:54874) 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.888+0000 7fb22a59c700 1 -- 172.21.15.106:0/1215510990 learned_addr learned my addr 172.21.15.106:0/1215510990 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.889+0000 7fb22a59c700 1 -- 172.21.15.106:0/1215510990 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c10aef0 con 0x7fb22c1085c0 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.889+0000 7fb22a59c700 1 --2- 172.21.15.106:0/1215510990 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c10a9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0098a0 tx=0x7fb21c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ea2b9c3403f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.889+0000 7fb22959a700 1 -- 172.21.15.106:0/1215510990 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb21c004160 con 0x7fb22c1085c0 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.889+0000 7fb22959a700 1 -- 172.21.15.106:0/1215510990 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb21c004300 con 0x7fb22c1085c0 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.889+0000 7fb22959a700 1 -- 172.21.15.106:0/1215510990 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb21c011b50 con 0x7fb22c1085c0 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.890+0000 7fb2308d6700 1 -- 172.21.15.106:0/1215510990 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 msgr2=0x7fb22c10a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.890+0000 7fb2308d6700 1 --2- 172.21.15.106:0/1215510990 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c10a9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0098a0 tx=0x7fb21c009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.890+0000 7fb2308d6700 1 -- 172.21.15.106:0/1215510990 shutdown_connections 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.890+0000 7fb2308d6700 1 --2- 172.21.15.106:0/1215510990 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c10a9b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:00.648 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.890+0000 7fb2308d6700 1 -- 172.21.15.106:0/1215510990 >> 172.21.15.106:0/1215510990 conn(0x7fb22c101ff0 msgr2=0x7fb22c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.890+0000 7fb2308d6700 1 -- 172.21.15.106:0/1215510990 shutdown_connections 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.890+0000 7fb2308d6700 1 -- 172.21.15.106:0/1215510990 wait complete. 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.890+0000 7fb2308d6700 1 Processor -- start 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.890+0000 7fb2308d6700 1 -- start start 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb2308d6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c19bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb2308d6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c10bf10 con 0x7fb22c1085c0 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb22a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c19bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb22a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c19bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54876/0 (socket says 172.21.15.106:54876) 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb22a59c700 1 -- 172.21.15.106:0/2007791673 learned_addr learned my addr 172.21.15.106:0/2007791673 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb22a59c700 1 -- 172.21.15.106:0/2007791673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c19c020 con 0x7fb22c1085c0 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb22a59c700 1 --2- 172.21.15.106:0/2007791673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c19bae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb21c011b60 tx=0x7fb21c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb21b7fe700 1 -- 172.21.15.106:0/2007791673 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb21c003970 con 0x7fb22c1085c0 2024-03-28T02:01:00.649 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb21b7fe700 1 -- 172.21.15.106:0/2007791673 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb21c003b10 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb2308d6700 1 -- 172.21.15.106:0/2007791673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c19c2b0 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb21b7fe700 1 -- 172.21.15.106:0/2007791673 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb21c024610 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.891+0000 7fb2308d6700 1 -- 172.21.15.106:0/2007791673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c19efa0 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.892+0000 7fb21b7fe700 1 -- 172.21.15.106:0/2007791673 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb21c024ca0 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.892+0000 7fb21b7fe700 1 -- 172.21.15.106:0/2007791673 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb21c011e00 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.892+0000 7fb2308d6700 1 -- 172.21.15.106:0/2007791673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22c069260 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.894+0000 7fb21b7fe700 1 -- 172.21.15.106:0/2007791673 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb21c01a9f0 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.966+0000 7fb2308d6700 1 -- 172.21.15.106:0/2007791673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb22c19f550 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.967+0000 7fb21b7fe700 1 -- 172.21.15.106:0/2007791673 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fb21c01a430 con 0x7fb22c1085c0 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.968+0000 7fb2308d6700 1 -- 172.21.15.106:0/2007791673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 msgr2=0x7fb22c19bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.968+0000 7fb2308d6700 1 --2- 172.21.15.106:0/2007791673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c19bae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb21c011b60 tx=0x7fb21c011c40 comp rx=0 tx=0).stop 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.968+0000 7fb2308d6700 1 -- 172.21.15.106:0/2007791673 shutdown_connections 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.968+0000 7fb2308d6700 1 --2- 172.21.15.106:0/2007791673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb22c1085c0 0x7fb22c19bae0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.968+0000 7fb2308d6700 1 -- 172.21.15.106:0/2007791673 >> 172.21.15.106:0/2007791673 conn(0x7fb22c101ff0 msgr2=0x7fb22c102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:00.650 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.968+0000 7fb2308d6700 1 -- 172.21.15.106:0/2007791673 shutdown_connections 2024-03-28T02:01:00.651 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:00:59.968+0000 7fb2308d6700 1 -- 172.21.15.106:0/2007791673 wait complete. 2024-03-28T02:01:00.651 INFO:teuthology.orchestra.run.smithi106.stdout:mgr not available, waiting (2/15)... 2024-03-28T02:01:01.067 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:00.820+0000 7f816c22d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:01:01.067 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:01.072+0000 7f816c22d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T02:01:01.334 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:01.303+0000 7f816c22d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T02:01:02.334 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:02 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:02.091+0000 7f816c22d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T02:01:02.882 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:02 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:02.661+0000 7f816c22d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T02:01:02.882 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:02 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:02.767+0000 7f816c22d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T02:01:03.166 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:02 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:02.994+0000 7f816c22d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T02:01:03.166 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: Activating manager daemon x 2024-03-28T02:01:03.166 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: mgrmap e2: x(active, starting, since 0.00401196s) 2024-03-28T02:01:03.166 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T02:01:03.166 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T02:01:03.167 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T02:01:03.167 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:01:03.167 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T02:01:03.167 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: Manager daemon x is now available 2024-03-28T02:01:03.167 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T02:01:03.167 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T02:01:03.167 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' 2024-03-28T02:01:03.167 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:03 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' 2024-03-28T02:01:03.767 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:01:03.767 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-28T02:01:03.767 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:01:03.767 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T02:01:03.767 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T02:01:03.767 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T02:01:03.769 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:03.769 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T02:01:03.769 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T02:01:03.769 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T02:01:03.776 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T02:01:03.776 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T02:01:03.776 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T02:01:03.776 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:03.776 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T02:01:03.776 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T02:01:03.777 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T02:00:42.806958+0000", 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:03.778 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.092+0000 7f02bd757700 1 Processor -- start 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.093+0000 7f02bd757700 1 -- start start 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.093+0000 7f02bd757700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.093+0000 7f02bd757700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b807be00 con 0x7f02b8108600 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.093+0000 7f02b6ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.093+0000 7f02b6ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54962/0 (socket says 172.21.15.106:54962) 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.093+0000 7f02b6ffd700 1 -- 172.21.15.106:0/2522381455 learned_addr learned my addr 172.21.15.106:0/2522381455 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.093+0000 7f02b6ffd700 1 -- 172.21.15.106:0/2522381455 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b810af30 con 0x7f02b8108600 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.093+0000 7f02b6ffd700 1 --2- 172.21.15.106:0/2522381455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b810a9f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0098a0 tx=0x7f02ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28076935dcfb7652 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.093+0000 7f02b5ffb700 1 -- 172.21.15.106:0/2522381455 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02ac004160 con 0x7f02b8108600 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.094+0000 7f02b5ffb700 1 -- 172.21.15.106:0/2522381455 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02ac004300 con 0x7f02b8108600 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.094+0000 7f02b5ffb700 1 -- 172.21.15.106:0/2522381455 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02ac011b50 con 0x7f02b8108600 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.094+0000 7f02bd757700 1 -- 172.21.15.106:0/2522381455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 msgr2=0x7f02b810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:03.779 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.094+0000 7f02bd757700 1 --2- 172.21.15.106:0/2522381455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b810a9f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0098a0 tx=0x7f02ac009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.094+0000 7f02bd757700 1 -- 172.21.15.106:0/2522381455 shutdown_connections 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.094+0000 7f02bd757700 1 --2- 172.21.15.106:0/2522381455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b810a9f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.094+0000 7f02bd757700 1 -- 172.21.15.106:0/2522381455 >> 172.21.15.106:0/2522381455 conn(0x7f02b8101ff0 msgr2=0x7f02b8104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.094+0000 7f02bd757700 1 -- 172.21.15.106:0/2522381455 shutdown_connections 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.094+0000 7f02bd757700 1 -- 172.21.15.106:0/2522381455 wait complete. 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.095+0000 7f02bd757700 1 Processor -- start 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.095+0000 7f02bd757700 1 -- start start 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.095+0000 7f02bd757700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b819baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.095+0000 7f02bd757700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b810bf00 con 0x7f02b8108600 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.095+0000 7f02b6ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b819baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.095+0000 7f02b6ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b819baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54976/0 (socket says 172.21.15.106:54976) 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.095+0000 7f02b6ffd700 1 -- 172.21.15.106:0/1010218771 learned_addr learned my addr 172.21.15.106:0/1010218771 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:03.780 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.096+0000 7f02b6ffd700 1 -- 172.21.15.106:0/1010218771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b819bfe0 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.096+0000 7f02b6ffd700 1 --2- 172.21.15.106:0/1010218771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b819baa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f02ac011670 tx=0x7f02ac011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.096+0000 7f029ffff700 1 -- 172.21.15.106:0/1010218771 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02ac0039c0 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.096+0000 7f02bd757700 1 -- 172.21.15.106:0/1010218771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b819c270 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.096+0000 7f029ffff700 1 -- 172.21.15.106:0/1010218771 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02ac011e50 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.096+0000 7f029ffff700 1 -- 172.21.15.106:0/1010218771 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02ac01a5c0 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.096+0000 7f02bd757700 1 -- 172.21.15.106:0/1010218771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b819ef60 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.097+0000 7f029ffff700 1 -- 172.21.15.106:0/1010218771 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 2) v1 ==== 45862+0+0 (secure 0 0 0) 0x7f02ac024430 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.097+0000 7f02bd757700 1 -- 172.21.15.106:0/1010218771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b8069260 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.097+0000 7f029ffff700 1 -- 172.21.15.106:0/1010218771 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f02ac04cd20 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.099+0000 7f029ffff700 1 -- 172.21.15.106:0/1010218771 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f02ac003b60 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.171+0000 7f02bd757700 1 -- 172.21.15.106:0/1010218771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f02b819f130 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.171+0000 7f029ffff700 1 -- 172.21.15.106:0/1010218771 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f02b819f130 con 0x7f02b8108600 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.172+0000 7f02bd757700 1 -- 172.21.15.106:0/1010218771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 msgr2=0x7f02b819baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.172+0000 7f02bd757700 1 --2- 172.21.15.106:0/1010218771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b819baa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f02ac011670 tx=0x7f02ac011b60 comp rx=0 tx=0).stop 2024-03-28T02:01:03.781 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.172+0000 7f02bd757700 1 -- 172.21.15.106:0/1010218771 shutdown_connections 2024-03-28T02:01:03.782 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.172+0000 7f02bd757700 1 --2- 172.21.15.106:0/1010218771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02b8108600 0x7f02b819baa0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:03.782 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.172+0000 7f02bd757700 1 -- 172.21.15.106:0/1010218771 >> 172.21.15.106:0/1010218771 conn(0x7f02b8101ff0 msgr2=0x7f02b8102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:03.782 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.173+0000 7f02bd757700 1 -- 172.21.15.106:0/1010218771 shutdown_connections 2024-03-28T02:01:03.782 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:03.173+0000 7f02bd757700 1 -- 172.21.15.106:0/1010218771 wait complete. 2024-03-28T02:01:03.782 INFO:teuthology.orchestra.run.smithi106.stdout:mgr not available, waiting (3/15)... 2024-03-28T02:01:04.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:04 smithi106 ceph-mon[31965]: from='mgr.14100 172.21.15.106:0/161012287' entity='mgr.x' 2024-03-28T02:01:04.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:04 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1010218771' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T02:01:04.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:04 smithi106 ceph-mon[31965]: mgrmap e3: x(active, since 1.00618s) 2024-03-28T02:01:06.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:06 smithi106 ceph-mon[31965]: mgrmap e4: x(active, since 2s) 2024-03-28T02:01:07.040 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:01:07.040 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-28T02:01:07.040 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-28T02:01:07.041 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T02:01:07.042 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:07.042 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T02:01:07.042 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T02:01:07.042 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:07.042 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T02:01:07.042 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:07.042 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T02:01:07.042 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T02:01:07.042 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T02:01:07.050 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T02:01:07.050 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T02:01:07.050 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T02:01:07.050 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:07.050 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T02:01:07.050 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T02:01:07.050 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T02:01:07.051 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T02:00:42.806958+0000", 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-28T02:01:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.123+0000 7fb256d45700 1 Processor -- start 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.123+0000 7fb256d45700 1 -- start start 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.123+0000 7fb256d45700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb250107f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.123+0000 7fb256d45700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb250108510 con 0x7fb250107b20 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.123+0000 7fb254ae1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb250107f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.123+0000 7fb254ae1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb250107f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54992/0 (socket says 172.21.15.106:54992) 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.123+0000 7fb254ae1700 1 -- 172.21.15.106:0/27598659 learned_addr learned my addr 172.21.15.106:0/27598659 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.124+0000 7fb254ae1700 1 -- 172.21.15.106:0/27598659 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250108d20 con 0x7fb250107b20 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.124+0000 7fb254ae1700 1 --2- 172.21.15.106:0/27598659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb250107f40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2440098a0 tx=0x7fb244009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fe998d6c2e86f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.124+0000 7fb24f7fe700 1 -- 172.21.15.106:0/27598659 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb244004160 con 0x7fb250107b20 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.124+0000 7fb24f7fe700 1 -- 172.21.15.106:0/27598659 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb244004300 con 0x7fb250107b20 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.124+0000 7fb24f7fe700 1 -- 172.21.15.106:0/27598659 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb244011b50 con 0x7fb250107b20 2024-03-28T02:01:07.053 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 -- 172.21.15.106:0/27598659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 msgr2=0x7fb250107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 --2- 172.21.15.106:0/27598659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb250107f40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2440098a0 tx=0x7fb244009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 -- 172.21.15.106:0/27598659 shutdown_connections 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 --2- 172.21.15.106:0/27598659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb250107f40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 -- 172.21.15.106:0/27598659 >> 172.21.15.106:0/27598659 conn(0x7fb2501030e0 msgr2=0x7fb250105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 -- 172.21.15.106:0/27598659 shutdown_connections 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 -- 172.21.15.106:0/27598659 wait complete. 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 Processor -- start 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 -- start start 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb25007db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb256d45700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25010c0d0 con 0x7fb250107b20 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.125+0000 7fb254ae1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb25007db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:07.054 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.126+0000 7fb254ae1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb25007db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54994/0 (socket says 172.21.15.106:54994) 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.126+0000 7fb254ae1700 1 -- 172.21.15.106:0/1772528615 learned_addr learned my addr 172.21.15.106:0/1772528615 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.126+0000 7fb254ae1700 1 -- 172.21.15.106:0/1772528615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25007e060 con 0x7fb250107b20 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.126+0000 7fb254ae1700 1 --2- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb25007db20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb244000c00 tx=0x7fb244011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.126+0000 7fb24dffb700 1 -- 172.21.15.106:0/1772528615 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb244003b20 con 0x7fb250107b20 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.126+0000 7fb24dffb700 1 -- 172.21.15.106:0/1772528615 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb244003cc0 con 0x7fb250107b20 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.126+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25007c250 con 0x7fb250107b20 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.126+0000 7fb24dffb700 1 -- 172.21.15.106:0/1772528615 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb244024910 con 0x7fb250107b20 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.126+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25007c790 con 0x7fb250107b20 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.127+0000 7fb24dffb700 1 -- 172.21.15.106:0/1772528615 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7fb244022020 con 0x7fb250107b20 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.127+0000 7fb24dffb700 1 --2- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.127+0000 7fb24dffb700 1 -- 172.21.15.106:0/1772528615 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb244011e00 con 0x7fb250107b20 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.127+0000 7fb24ffff700 1 --2- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:07.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.127+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb250069260 con 0x7fb250107b20 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.128+0000 7fb24ffff700 1 --2- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7fb240038ed0 0x7fb24003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb23c006fd0 tx=0x7fb23c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.133+0000 7fb24dffb700 1 -- 172.21.15.106:0/1772528615 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb2440172b0 con 0x7fb250107b20 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.404+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb25007ce40 con 0x7fb250107b20 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.404+0000 7fb24dffb700 1 -- 172.21.15.106:0/1772528615 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fb24402cdf0 con 0x7fb250107b20 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7fb240038ed0 msgr2=0x7fb24003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 --2- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7fb240038ed0 0x7fb24003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb23c006fd0 tx=0x7fb23c006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 msgr2=0x7fb25007db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 --2- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb25007db20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb244000c00 tx=0x7fb244011c10 comp rx=0 tx=0).stop 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 shutdown_connections 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 --2- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 --2- 172.21.15.106:0/1772528615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb250107b20 0x7fb25007db20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:07.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 >> 172.21.15.106:0/1772528615 conn(0x7fb2501030e0 msgr2=0x7fb250103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:07.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 shutdown_connections 2024-03-28T02:01:07.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:06.407+0000 7fb256d45700 1 -- 172.21.15.106:0/1772528615 wait complete. 2024-03-28T02:01:07.057 INFO:teuthology.orchestra.run.smithi106.stdout:mgr is available 2024-03-28T02:01:07.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:07 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1772528615' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T02:01:08.141 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:08 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2893772493' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout fsid = d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-28T02:01:08.151 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.409+0000 7f0a69e61700 1 Processor -- start 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.409+0000 7f0a69e61700 1 -- start start 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.409+0000 7f0a69e61700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a64107f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.409+0000 7f0a69e61700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a64108510 con 0x7f0a64107b20 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.409+0000 7f0a637fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a64107f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.409+0000 7f0a637fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a64107f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54998/0 (socket says 172.21.15.106:54998) 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.409+0000 7f0a637fe700 1 -- 172.21.15.106:0/4082580326 learned_addr learned my addr 172.21.15.106:0/4082580326 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.409+0000 7f0a637fe700 1 -- 172.21.15.106:0/4082580326 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a64108d20 con 0x7f0a64107b20 2024-03-28T02:01:08.152 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.409+0000 7f0a637fe700 1 --2- 172.21.15.106:0/4082580326 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a64107f40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c0098a0 tx=0x7f0a5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa955d4dbe0322a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a627fc700 1 -- 172.21.15.106:0/4082580326 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a5c004160 con 0x7f0a64107b20 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a627fc700 1 -- 172.21.15.106:0/4082580326 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a5c004300 con 0x7f0a64107b20 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a627fc700 1 -- 172.21.15.106:0/4082580326 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a5c011b50 con 0x7f0a64107b20 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a69e61700 1 -- 172.21.15.106:0/4082580326 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 msgr2=0x7f0a64107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a69e61700 1 --2- 172.21.15.106:0/4082580326 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a64107f40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c0098a0 tx=0x7f0a5c009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a69e61700 1 -- 172.21.15.106:0/4082580326 shutdown_connections 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a69e61700 1 --2- 172.21.15.106:0/4082580326 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a64107f40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a69e61700 1 -- 172.21.15.106:0/4082580326 >> 172.21.15.106:0/4082580326 conn(0x7f0a641030e0 msgr2=0x7f0a64105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a69e61700 1 -- 172.21.15.106:0/4082580326 shutdown_connections 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.410+0000 7f0a69e61700 1 -- 172.21.15.106:0/4082580326 wait complete. 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.411+0000 7f0a69e61700 1 Processor -- start 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.411+0000 7f0a69e61700 1 -- start start 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.411+0000 7f0a69e61700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a6419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.411+0000 7f0a69e61700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6410c0d0 con 0x7f0a64107b20 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.411+0000 7f0a637fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a6419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:08.153 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.411+0000 7f0a637fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a6419bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55014/0 (socket says 172.21.15.106:55014) 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.411+0000 7f0a637fe700 1 -- 172.21.15.106:0/2893772493 learned_addr learned my addr 172.21.15.106:0/2893772493 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.412+0000 7f0a637fe700 1 -- 172.21.15.106:0/2893772493 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6419c000 con 0x7f0a64107b20 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.412+0000 7f0a637fe700 1 --2- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a6419bac0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c011670 tx=0x7f0a5c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.412+0000 7f0a60ff9700 1 -- 172.21.15.106:0/2893772493 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a5c0039c0 con 0x7f0a64107b20 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.412+0000 7f0a60ff9700 1 -- 172.21.15.106:0/2893772493 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a5c011e50 con 0x7f0a64107b20 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.412+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6419c290 con 0x7f0a64107b20 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.412+0000 7f0a60ff9700 1 -- 172.21.15.106:0/2893772493 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a5c01a5c0 con 0x7f0a64107b20 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.412+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6419ef80 con 0x7f0a64107b20 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.413+0000 7f0a60ff9700 1 -- 172.21.15.106:0/2893772493 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f0a5c01a870 con 0x7f0a64107b20 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.413+0000 7f0a60ff9700 1 --2- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f0a50038ed0 0x7f0a5003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.413+0000 7f0a60ff9700 1 -- 172.21.15.106:0/2893772493 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0a5c04e0b0 con 0x7f0a64107b20 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.413+0000 7f0a62ffd700 1 --2- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f0a50038ed0 0x7f0a5003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:08.154 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.413+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a64069260 con 0x7f0a64107b20 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.413+0000 7f0a62ffd700 1 --2- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f0a50038ed0 0x7f0a5003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c006fd0 tx=0x7f0a4c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.418+0000 7f0a60ff9700 1 -- 172.21.15.106:0/2893772493 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a5c019c70 con 0x7f0a64107b20 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.610+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0a6419f150 con 0x7f0a64107b20 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.611+0000 7f0a60ff9700 1 -- 172.21.15.106:0/2893772493 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f0a5c020240 con 0x7f0a64107b20 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f0a50038ed0 msgr2=0x7f0a5003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 --2- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f0a50038ed0 0x7f0a5003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c006fd0 tx=0x7f0a4c006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 msgr2=0x7f0a6419bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 --2- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a6419bac0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c011670 tx=0x7f0a5c011b60 comp rx=0 tx=0).stop 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 shutdown_connections 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 --2- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f0a50038ed0 0x7f0a5003b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 --2- 172.21.15.106:0/2893772493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a64107b20 0x7f0a6419bac0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 >> 172.21.15.106:0/2893772493 conn(0x7f0a641030e0 msgr2=0x7f0a64103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 shutdown_connections 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:07.613+0000 7f0a69e61700 1 -- 172.21.15.106:0/2893772493 wait complete. 2024-03-28T02:01:08.155 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling cephadm module... 2024-03-28T02:01:09.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:09 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2875330872' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T02:01:09.334 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: ignoring --setuser ceph since I am not root 2024-03-28T02:01:09.334 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: ignoring --setgroup ceph since I am not root 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.524+0000 7f24d8ccd700 1 Processor -- start 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.524+0000 7f24d8ccd700 1 -- start start 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.524+0000 7f24d8ccd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d4106e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.524+0000 7f24d8ccd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d4107420 con 0x7f24d4106a30 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.525+0000 7f24d259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d4106e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.525+0000 7f24d259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d4106e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39548/0 (socket says 172.21.15.106:39548) 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.525+0000 7f24d259c700 1 -- 172.21.15.106:0/1767187955 learned_addr learned my addr 172.21.15.106:0/1767187955 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.525+0000 7f24d259c700 1 -- 172.21.15.106:0/1767187955 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d4107c30 con 0x7f24d4106a30 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.525+0000 7f24d259c700 1 --2- 172.21.15.106:0/1767187955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d4106e50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f24c80098a0 tx=0x7f24c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06ac9a2e410187 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.525+0000 7f24d159a700 1 -- 172.21.15.106:0/1767187955 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24c8004160 con 0x7f24d4106a30 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.525+0000 7f24d159a700 1 -- 172.21.15.106:0/1767187955 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24c8004300 con 0x7f24d4106a30 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.525+0000 7f24d159a700 1 -- 172.21.15.106:0/1767187955 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24c8011b50 con 0x7f24d4106a30 2024-03-28T02:01:09.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.526+0000 7f24d8ccd700 1 -- 172.21.15.106:0/1767187955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 msgr2=0x7f24d4106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.526+0000 7f24d8ccd700 1 --2- 172.21.15.106:0/1767187955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d4106e50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f24c80098a0 tx=0x7f24c8009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.526+0000 7f24d8ccd700 1 -- 172.21.15.106:0/1767187955 shutdown_connections 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.526+0000 7f24d8ccd700 1 --2- 172.21.15.106:0/1767187955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d4106e50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.526+0000 7f24d8ccd700 1 -- 172.21.15.106:0/1767187955 >> 172.21.15.106:0/1767187955 conn(0x7f24d4101ff0 msgr2=0x7f24d4104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.526+0000 7f24d8ccd700 1 -- 172.21.15.106:0/1767187955 shutdown_connections 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.526+0000 7f24d8ccd700 1 -- 172.21.15.106:0/1767187955 wait complete. 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d8ccd700 1 Processor -- start 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d8ccd700 1 -- start start 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d8ccd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d8ccd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d410afe0 con 0x7f24d4106a30 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d419e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39558/0 (socket says 172.21.15.106:39558) 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d259c700 1 -- 172.21.15.106:0/2875330872 learned_addr learned my addr 172.21.15.106:0/2875330872 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:09.565 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d259c700 1 -- 172.21.15.106:0/2875330872 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d407bbe0 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d259c700 1 --2- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d419e820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24c8011670 tx=0x7f24c8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24c37fe700 1 -- 172.21.15.106:0/2875330872 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24c80039c0 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24c37fe700 1 -- 172.21.15.106:0/2875330872 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24c8011e50 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.527+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d407be70 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.528+0000 7f24c37fe700 1 -- 172.21.15.106:0/2875330872 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24c801a5c0 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.528+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d419efb0 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.528+0000 7f24c37fe700 1 -- 172.21.15.106:0/2875330872 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f24c801a870 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.528+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d40637c0 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.529+0000 7f24c37fe700 1 --2- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f24bc038ed0 0x7f24bc03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.529+0000 7f24c37fe700 1 -- 172.21.15.106:0/2875330872 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f24c804d0c0 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.529+0000 7f24d1d9b700 1 --2- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f24bc038ed0 0x7f24bc03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.529+0000 7f24d1d9b700 1 --2- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f24bc038ed0 0x7f24bc03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f24c4006fd0 tx=0x7f24c4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.533+0000 7f24c37fe700 1 -- 172.21.15.106:0/2875330872 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f24c8019c70 con 0x7f24d4106a30 2024-03-28T02:01:09.566 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:08.782+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f24d406a4c0 con 0x7f24d4106a30 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.071+0000 7f24c37fe700 1 -- 172.21.15.106:0/2875330872 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f24c8020240 con 0x7f24d4106a30 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.073+0000 7f24c37fe700 1 -- 172.21.15.106:0/2875330872 <== mon.0 v2:172.21.15.106:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f24c802d910 con 0x7f24d4106a30 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f24bc038ed0 msgr2=0x7f24bc03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 --2- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f24bc038ed0 0x7f24bc03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f24c4006fd0 tx=0x7f24c4006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 msgr2=0x7f24d419e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 --2- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d419e820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24c8011670 tx=0x7f24c8011b60 comp rx=0 tx=0).stop 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 shutdown_connections 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 --2- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f24bc038ed0 0x7f24bc03b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 --2- 172.21.15.106:0/2875330872 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24d4106a30 0x7f24d419e820 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 >> 172.21.15.106:0/2875330872 conn(0x7f24d4101ff0 msgr2=0x7f24d4102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 shutdown_connections 2024-03-28T02:01:09.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:09.074+0000 7f24d8ccd700 1 -- 172.21.15.106:0/2875330872 wait complete. 2024-03-28T02:01:09.681 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:09.631+0000 7f612af58200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T02:01:10.001 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:09.726+0000 7f612af58200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T02:01:10.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:10 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2875330872' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T02:01:10.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:10 smithi106 ceph-mon[31965]: mgrmap e5: x(active, since 6s) 2024-03-28T02:01:10.584 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:10.384+0000 7f612af58200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T02:01:10.584 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:10.475+0000 7f612af58200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T02:01:10.830 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-28T02:01:10.830 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T02:01:10.830 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T02:01:10.830 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.022+0000 7f1d4c68e700 1 Processor -- start 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.022+0000 7f1d4c68e700 1 -- start start 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.022+0000 7f1d4c68e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d44106830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.022+0000 7f1d4c68e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4407bde0 con 0x7f1d44106410 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.022+0000 7f1d4a42a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d44106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.022+0000 7f1d4a42a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d44106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39592/0 (socket says 172.21.15.106:39592) 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.023+0000 7f1d4a42a700 1 -- 172.21.15.106:0/1181816554 learned_addr learned my addr 172.21.15.106:0/1181816554 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.023+0000 7f1d4a42a700 1 -- 172.21.15.106:0/1181816554 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44106d70 con 0x7f1d44106410 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.023+0000 7f1d4a42a700 1 --2- 172.21.15.106:0/1181816554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d44106830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1d340098a0 tx=0x7f1d34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e2c095281a3ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.023+0000 7f1d49428700 1 -- 172.21.15.106:0/1181816554 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d34004160 con 0x7f1d44106410 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.023+0000 7f1d49428700 1 -- 172.21.15.106:0/1181816554 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d34004300 con 0x7f1d44106410 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.023+0000 7f1d49428700 1 -- 172.21.15.106:0/1181816554 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d34011b50 con 0x7f1d44106410 2024-03-28T02:01:10.831 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.024+0000 7f1d4c68e700 1 -- 172.21.15.106:0/1181816554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 msgr2=0x7f1d44106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.024+0000 7f1d4c68e700 1 --2- 172.21.15.106:0/1181816554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d44106830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1d340098a0 tx=0x7f1d34009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.024+0000 7f1d4c68e700 1 -- 172.21.15.106:0/1181816554 shutdown_connections 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.024+0000 7f1d4c68e700 1 --2- 172.21.15.106:0/1181816554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d44106830 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.024+0000 7f1d4c68e700 1 -- 172.21.15.106:0/1181816554 >> 172.21.15.106:0/1181816554 conn(0x7f1d44101ff0 msgr2=0x7f1d44104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.024+0000 7f1d4c68e700 1 -- 172.21.15.106:0/1181816554 shutdown_connections 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.024+0000 7f1d4c68e700 1 -- 172.21.15.106:0/1181816554 wait complete. 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.025+0000 7f1d4c68e700 1 Processor -- start 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.025+0000 7f1d4c68e700 1 -- start start 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.025+0000 7f1d4c68e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d4419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.025+0000 7f1d4c68e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d44107e00 con 0x7f1d44106410 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.025+0000 7f1d4a42a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d4419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.025+0000 7f1d4a42a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d4419dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39600/0 (socket says 172.21.15.106:39600) 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.025+0000 7f1d4a42a700 1 -- 172.21.15.106:0/2754292248 learned_addr learned my addr 172.21.15.106:0/2754292248 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.025+0000 7f1d4a42a700 1 -- 172.21.15.106:0/2754292248 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4419e1f0 con 0x7f1d44106410 2024-03-28T02:01:10.832 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.026+0000 7f1d4a42a700 1 --2- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d4419dcb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d34011b60 tx=0x7f1d34011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.026+0000 7f1d3b7fe700 1 -- 172.21.15.106:0/2754292248 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d34003b20 con 0x7f1d44106410 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.026+0000 7f1d3b7fe700 1 -- 172.21.15.106:0/2754292248 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d34003cc0 con 0x7f1d44106410 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.026+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4419e480 con 0x7f1d44106410 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.026+0000 7f1d3b7fe700 1 -- 172.21.15.106:0/2754292248 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d34024840 con 0x7f1d44106410 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.026+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d441a1170 con 0x7f1d44106410 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.027+0000 7f1d3b7fe700 1 -- 172.21.15.106:0/2754292248 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1d34022020 con 0x7f1d44106410 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.027+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d44069260 con 0x7f1d44106410 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.027+0000 7f1d3b7fe700 1 --2- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f1d30038f70 0x7f1d3003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.027+0000 7f1d3b7fe700 1 -- 172.21.15.106:0/2754292248 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1d34011e00 con 0x7f1d44106410 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.027+0000 7f1d49c29700 1 -- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f1d30038f70 msgr2=0x7f1d3003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/1510653090 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.027+0000 7f1d49c29700 1 --2- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f1d30038f70 0x7f1d3003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.032+0000 7f1d3b7fe700 1 -- 172.21.15.106:0/2754292248 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d3400b3e0 con 0x7f1d44106410 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.228+0000 7f1d49c29700 1 -- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f1d30038f70 msgr2=0x7f1d3003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/1510653090 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.228+0000 7f1d49c29700 1 --2- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f1d30038f70 0x7f1d3003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T02:01:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.286+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1d440008d0 con 0x7f1d44106410 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.286+0000 7f1d3b7fe700 1 -- 172.21.15.106:0/2754292248 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f1d34016260 con 0x7f1d44106410 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.288+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f1d30038f70 msgr2=0x7f1d3003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.288+0000 7f1d4c68e700 1 --2- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f1d30038f70 0x7f1d3003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.288+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 msgr2=0x7f1d4419dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.288+0000 7f1d4c68e700 1 --2- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d4419dcb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d34011b60 tx=0x7f1d34011c40 comp rx=0 tx=0).stop 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.288+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 shutdown_connections 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.288+0000 7f1d4c68e700 1 --2- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f1d30038f70 0x7f1d3003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.288+0000 7f1d4c68e700 1 --2- 172.21.15.106:0/2754292248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d44106410 0x7f1d4419dcb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.288+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 >> 172.21.15.106:0/2754292248 conn(0x7f1d44101ff0 msgr2=0x7f1d441948d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.289+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 shutdown_connections 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:10.289+0000 7f1d4c68e700 1 -- 172.21.15.106:0/2754292248 wait complete. 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for the mgr to restart... 2024-03-28T02:01:10.834 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr epoch 5... 2024-03-28T02:01:10.922 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:10.617+0000 7f612af58200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T02:01:10.922 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:10.711+0000 7f612af58200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T02:01:10.922 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:10.795+0000 7f612af58200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T02:01:11.177 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:11 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2754292248' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T02:01:11.177 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:11.071+0000 7f612af58200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T02:01:11.178 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:11.183+0000 7f612af58200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T02:01:11.533 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:11.272+0000 7f612af58200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T02:01:11.533 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:11.365+0000 7f612af58200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T02:01:11.813 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:11.538+0000 7f612af58200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T02:01:12.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:12.478+0000 7f612af58200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T02:01:12.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:12.605+0000 7f612af58200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T02:01:13.226 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:12.933+0000 7f612af58200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T02:01:13.227 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:13.020+0000 7f612af58200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T02:01:13.227 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:13.233+0000 7f612af58200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:01:13.584 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:13.414+0000 7f612af58200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T02:01:13.946 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:13.700+0000 7f612af58200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:01:13.946 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:13.952+0000 7f612af58200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T02:01:14.216 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:14.185+0000 7f612af58200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T02:01:15.334 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:14.971+0000 7f612af58200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T02:01:15.833 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:15.530+0000 7f612af58200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T02:01:15.833 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:15.626+0000 7f612af58200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T02:01:15.833 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:15.839+0000 7f612af58200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T02:01:16.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: Active manager daemon x restarted 2024-03-28T02:01:16.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: Activating manager daemon x 2024-03-28T02:01:16.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T02:01:16.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: mgrmap e6: x(active, starting, since 0.00353464s) 2024-03-28T02:01:16.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:01:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T02:01:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T02:01:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T02:01:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T02:01:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: Manager daemon x is now available 2024-03-28T02:01:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:15 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:17.211 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:16 smithi106 ceph-mon[31965]: Found migration_current of "None". Setting to last migration. 2024-03-28T02:01:17.211 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:16 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:17.211 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:16 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T02:01:17.211 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:16 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T02:01:17.211 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:16 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:17.211 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:16 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:17.211 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:16 smithi106 ceph-mon[31965]: mgrmap e7: x(active, since 1.0055s) 2024-03-28T02:01:17.462 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-28T02:01:17.462 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.212+0000 7f9ccd1b4700 1 Processor -- start 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.212+0000 7f9ccd1b4700 1 -- start start 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.212+0000 7f9ccd1b4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc810a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.212+0000 7f9ccd1b4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc807be60 con 0x7f9cc81085a0 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.212+0000 7f9cc6d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc810a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.212+0000 7f9cc6d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc810a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39608/0 (socket says 172.21.15.106:39608) 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.212+0000 7f9cc6d9d700 1 -- 172.21.15.106:0/2264050438 learned_addr learned my addr 172.21.15.106:0/2264050438 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.213+0000 7f9cc6d9d700 1 -- 172.21.15.106:0/2264050438 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc810aed0 con 0x7f9cc81085a0 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.213+0000 7f9cc6d9d700 1 --2- 172.21.15.106:0/2264050438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc810a990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80098a0 tx=0x7f9cb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce305fcc5b04ba9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.213+0000 7f9cc5d9b700 1 -- 172.21.15.106:0/2264050438 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cb8004160 con 0x7f9cc81085a0 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.213+0000 7f9cc5d9b700 1 -- 172.21.15.106:0/2264050438 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cb8004300 con 0x7f9cc81085a0 2024-03-28T02:01:17.463 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.213+0000 7f9cc5d9b700 1 -- 172.21.15.106:0/2264050438 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cb8011b50 con 0x7f9cc81085a0 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.214+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2264050438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 msgr2=0x7f9cc810a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.214+0000 7f9ccd1b4700 1 --2- 172.21.15.106:0/2264050438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc810a990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80098a0 tx=0x7f9cb8009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.214+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2264050438 shutdown_connections 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.214+0000 7f9ccd1b4700 1 --2- 172.21.15.106:0/2264050438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc810a990 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.214+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2264050438 >> 172.21.15.106:0/2264050438 conn(0x7f9cc8101ff0 msgr2=0x7f9cc8104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.214+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2264050438 shutdown_connections 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.214+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2264050438 wait complete. 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.214+0000 7f9ccd1b4700 1 Processor -- start 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.215+0000 7f9ccd1b4700 1 -- start start 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.215+0000 7f9ccd1b4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc819baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.215+0000 7f9ccd1b4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc810bf00 con 0x7f9cc81085a0 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.215+0000 7f9cc6d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc819baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.215+0000 7f9cc6d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc819baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39612/0 (socket says 172.21.15.106:39612) 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.215+0000 7f9cc6d9d700 1 -- 172.21.15.106:0/2490190816 learned_addr learned my addr 172.21.15.106:0/2490190816 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.215+0000 7f9cc6d9d700 1 -- 172.21.15.106:0/2490190816 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc819bfe0 con 0x7f9cc81085a0 2024-03-28T02:01:17.464 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.215+0000 7f9cc6d9d700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc819baa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8011670 tx=0x7f9cb8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.216+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cb80039c0 con 0x7f9cc81085a0 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.216+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2490190816 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc819c270 con 0x7f9cc81085a0 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.216+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cb8011e50 con 0x7f9cc81085a0 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.216+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2490190816 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc819ef60 con 0x7f9cc81085a0 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.216+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cb801a5c0 con 0x7f9cc81085a0 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.216+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9cb801a8c0 con 0x7f9cc81085a0 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.217+0000 7f9caffff700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 0x7f9cb003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.217+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 --> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9cb003baa0 con 0x7f9cb0038ed0 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.217+0000 7f9cc659c700 1 -- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 msgr2=0x7f9cb003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/1510653090 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.217+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9cb804d480 con 0x7f9cc81085a0 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.217+0000 7f9cc659c700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 0x7f9cb003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.418+0000 7f9cc659c700 1 -- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 msgr2=0x7f9cb003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/1510653090 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.418+0000 7f9cc659c700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 0x7f9cb003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.819+0000 7f9cc659c700 1 -- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 msgr2=0x7f9cb003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/1510653090 2024-03-28T02:01:17.465 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:11.819+0000 7f9cc659c700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 0x7f9cb003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:12.620+0000 7f9cc659c700 1 -- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 msgr2=0x7f9cb003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/1510653090 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:12.620+0000 7f9cc659c700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 0x7f9cb003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:14.222+0000 7f9cc659c700 1 -- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 msgr2=0x7f9cb003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/1510653090 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:14.222+0000 7f9cc659c700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 0x7f9cb003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:15.854+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f9cb801abb0 con 0x7f9cc81085a0 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:15.854+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 msgr2=0x7f9cb003b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:15.854+0000 7f9caffff700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 0x7f9cb003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.855+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f9cb804edc0 con 0x7f9cc81085a0 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.855+0000 7f9caffff700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f9cb003c7c0 0x7f9cb003ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.855+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9cb801bdf0 con 0x7f9cb003c7c0 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.855+0000 7f9cc659c700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f9cb003c7c0 0x7f9cb003ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.856+0000 7f9cc659c700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f9cb003c7c0 0x7f9cb003ebb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc003a10 tx=0x7f9cbc009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.857+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 <== mgr.14120 v2:172.21.15.106:6800/305853075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f9cb801bdf0 con 0x7f9cb003c7c0 2024-03-28T02:01:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.863+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2490190816 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9cc8069260 con 0x7f9cb003c7c0 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9caffff700 1 -- 172.21.15.106:0/2490190816 <== mgr.14120 v2:172.21.15.106:6800/305853075 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9cc8069260 con 0x7f9cb003c7c0 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f9cb003c7c0 msgr2=0x7f9cb003ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f9cb003c7c0 0x7f9cb003ebb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc003a10 tx=0x7f9cbc009290 comp rx=0 tx=0).stop 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 msgr2=0x7f9cc819baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc819baa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8011670 tx=0x7f9cb8011b60 comp rx=0 tx=0).stop 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2490190816 shutdown_connections 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f9cb003c7c0 0x7f9cb003ebb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc81085a0 0x7f9cc819baa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 --2- 172.21.15.106:0/2490190816 >> [v2:172.21.15.106:6800/1510653090,v1:172.21.15.106:6801/1510653090] conn(0x7f9cb0038ed0 0x7f9cb003b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2490190816 >> 172.21.15.106:0/2490190816 conn(0x7f9cc8101ff0 msgr2=0x7f9cc8102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2490190816 shutdown_connections 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:16.864+0000 7f9ccd1b4700 1 -- 172.21.15.106:0/2490190816 wait complete. 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:mgr epoch 5 is available 2024-03-28T02:01:17.467 INFO:teuthology.orchestra.run.smithi106.stdout:Setting orchestrator backend to cephadm... 2024-03-28T02:01:18.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:17 smithi106 ceph-mon[31965]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T02:01:18.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:17 smithi106 ceph-mon[31965]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T02:01:18.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:17 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:18.684 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.860+0000 7fd78aeb6700 1 Processor -- start 2024-03-28T02:01:18.684 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd78aeb6700 1 -- start start 2024-03-28T02:01:18.684 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd78aeb6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd784112910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:18.684 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd78aeb6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd784112ee0 con 0x7fd7841124f0 2024-03-28T02:01:18.684 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd788c52700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd784112910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:18.684 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd788c52700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd784112910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40492/0 (socket says 172.21.15.106:40492) 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd788c52700 1 -- 172.21.15.106:0/968687590 learned_addr learned my addr 172.21.15.106:0/968687590 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd788c52700 1 -- 172.21.15.106:0/968687590 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7841136f0 con 0x7fd7841124f0 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd788c52700 1 --2- 172.21.15.106:0/968687590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd784112910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7780098a0 tx=0x7fd778009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62be9bfc94ecfa05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd7837fe700 1 -- 172.21.15.106:0/968687590 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd778004160 con 0x7fd7841124f0 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd7837fe700 1 -- 172.21.15.106:0/968687590 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd778004300 con 0x7fd7841124f0 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.861+0000 7fd7837fe700 1 -- 172.21.15.106:0/968687590 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd778011b50 con 0x7fd7841124f0 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.862+0000 7fd78aeb6700 1 -- 172.21.15.106:0/968687590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 msgr2=0x7fd784112910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.862+0000 7fd78aeb6700 1 --2- 172.21.15.106:0/968687590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd784112910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7780098a0 tx=0x7fd778009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.862+0000 7fd78aeb6700 1 -- 172.21.15.106:0/968687590 shutdown_connections 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.862+0000 7fd78aeb6700 1 --2- 172.21.15.106:0/968687590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd784112910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.862+0000 7fd78aeb6700 1 -- 172.21.15.106:0/968687590 >> 172.21.15.106:0/968687590 conn(0x7fd78406cbf0 msgr2=0x7fd78406d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.862+0000 7fd78aeb6700 1 -- 172.21.15.106:0/968687590 shutdown_connections 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.862+0000 7fd78aeb6700 1 -- 172.21.15.106:0/968687590 wait complete. 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd78aeb6700 1 Processor -- start 2024-03-28T02:01:18.685 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd78aeb6700 1 -- start start 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd78aeb6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd7841a6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd78aeb6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd784116aa0 con 0x7fd7841124f0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd788c52700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd7841a6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd788c52700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd7841a6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40498/0 (socket says 172.21.15.106:40498) 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd788c52700 1 -- 172.21.15.106:0/2962050897 learned_addr learned my addr 172.21.15.106:0/2962050897 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd788c52700 1 -- 172.21.15.106:0/2962050897 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7841a6b70 con 0x7fd7841124f0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd788c52700 1 --2- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd7841a6630 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd778011850 tx=0x7fd778011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd781ffb700 1 -- 172.21.15.106:0/2962050897 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd778003af0 con 0x7fd7841124f0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd781ffb700 1 -- 172.21.15.106:0/2962050897 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd778020070 con 0x7fd7841124f0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd781ffb700 1 -- 172.21.15.106:0/2962050897 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7780195a0 con 0x7fd7841124f0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.863+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7841a6e00 con 0x7fd7841124f0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.864+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7841a7920 con 0x7fd7841124f0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.864+0000 7fd781ffb700 1 -- 172.21.15.106:0/2962050897 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fd778019850 con 0x7fd7841124f0 2024-03-28T02:01:18.686 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.865+0000 7fd781ffb700 1 --2- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fd774038e50 0x7fd77403b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.865+0000 7fd781ffb700 1 -- 172.21.15.106:0/2962050897 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd778011e00 con 0x7fd7841124f0 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.865+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7840637c0 con 0x7fd7841124f0 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.865+0000 7fd783fff700 1 --2- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fd774038e50 0x7fd77403b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.865+0000 7fd783fff700 1 --2- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fd774038e50 0x7fd77403b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd770006fd0 tx=0x7fd770006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.870+0000 7fd781ffb700 1 -- 172.21.15.106:0/2962050897 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd778003c90 con 0x7fd7841124f0 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:17.907+0000 7fd781ffb700 1 -- 172.21.15.106:0/2962050897 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fd77802bd00 con 0x7fd7841124f0 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.081+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fd784110530 con 0x7fd774038e50 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.090+0000 7fd781ffb700 1 -- 172.21.15.106:0/2962050897 <== mgr.14120 v2:172.21.15.106:6800/305853075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd784110530 con 0x7fd774038e50 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fd774038e50 msgr2=0x7fd77403b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 --2- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fd774038e50 0x7fd77403b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd770006fd0 tx=0x7fd770006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 msgr2=0x7fd7841a6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 --2- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd7841a6630 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd778011850 tx=0x7fd778011c10 comp rx=0 tx=0).stop 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 shutdown_connections 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 --2- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fd774038e50 0x7fd77403b310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:18.687 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 --2- 172.21.15.106:0/2962050897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7841124f0 0x7fd7841a6630 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:18.688 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 >> 172.21.15.106:0/2962050897 conn(0x7fd78406cbf0 msgr2=0x7fd78410fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:18.688 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 shutdown_connections 2024-03-28T02:01:18.688 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:18.092+0000 7fd78aeb6700 1 -- 172.21.15.106:0/2962050897 wait complete. 2024-03-28T02:01:18.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:18 smithi106 ceph-mon[31965]: [28/Mar/2024:02:01:17] ENGINE Bus STARTING 2024-03-28T02:01:18.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:18 smithi106 ceph-mon[31965]: [28/Mar/2024:02:01:17] ENGINE Serving on http://172.21.15.106:8765 2024-03-28T02:01:18.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:18 smithi106 ceph-mon[31965]: [28/Mar/2024:02:01:17] ENGINE Serving on https://172.21.15.106:7150 2024-03-28T02:01:18.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:18 smithi106 ceph-mon[31965]: [28/Mar/2024:02:01:17] ENGINE Bus STARTED 2024-03-28T02:01:18.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:18 smithi106 ceph-mon[31965]: mgrmap e8: x(active, since 2s) 2024-03-28T02:01:18.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:18 smithi106 ceph-mon[31965]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:18.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:18 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:18.981 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:18 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.083+0000 7f86a762a700 1 Processor -- start 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.083+0000 7f86a762a700 1 -- start start 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f86a762a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a0107f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f86a762a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0108510 con 0x7f86a0107b20 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f86a53c6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a0107f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f86a53c6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a0107f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40504/0 (socket says 172.21.15.106:40504) 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f86a53c6700 1 -- 172.21.15.106:0/3421519219 learned_addr learned my addr 172.21.15.106:0/3421519219 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f86a53c6700 1 -- 172.21.15.106:0/3421519219 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0108d20 con 0x7f86a0107b20 2024-03-28T02:01:19.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f86a53c6700 1 --2- 172.21.15.106:0/3421519219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a0107f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f869c0098a0 tx=0x7f869c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97c598f9fc90d05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f8697fff700 1 -- 172.21.15.106:0/3421519219 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f869c004160 con 0x7f86a0107b20 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f8697fff700 1 -- 172.21.15.106:0/3421519219 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f869c004300 con 0x7f86a0107b20 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.084+0000 7f8697fff700 1 -- 172.21.15.106:0/3421519219 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f869c011b50 con 0x7f86a0107b20 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.085+0000 7f86a762a700 1 -- 172.21.15.106:0/3421519219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 msgr2=0x7f86a0107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.085+0000 7f86a762a700 1 --2- 172.21.15.106:0/3421519219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a0107f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f869c0098a0 tx=0x7f869c009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.085+0000 7f86a762a700 1 -- 172.21.15.106:0/3421519219 shutdown_connections 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.085+0000 7f86a762a700 1 --2- 172.21.15.106:0/3421519219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a0107f40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.085+0000 7f86a762a700 1 -- 172.21.15.106:0/3421519219 >> 172.21.15.106:0/3421519219 conn(0x7f86a01030e0 msgr2=0x7f86a0105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.085+0000 7f86a762a700 1 -- 172.21.15.106:0/3421519219 shutdown_connections 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.085+0000 7f86a762a700 1 -- 172.21.15.106:0/3421519219 wait complete. 2024-03-28T02:01:19.825 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a762a700 1 Processor -- start 2024-03-28T02:01:19.826 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a762a700 1 -- start start 2024-03-28T02:01:19.826 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a762a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a019bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:19.826 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a762a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a010c0d0 con 0x7f86a0107b20 2024-03-28T02:01:19.826 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a53c6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a019bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:19.826 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a53c6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a019bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40518/0 (socket says 172.21.15.106:40518) 2024-03-28T02:01:19.826 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a53c6700 1 -- 172.21.15.106:0/1817999894 learned_addr learned my addr 172.21.15.106:0/1817999894 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:19.826 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a53c6700 1 -- 172.21.15.106:0/1817999894 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a019c000 con 0x7f86a0107b20 2024-03-28T02:01:19.826 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a53c6700 1 --2- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a019bac0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f869c011670 tx=0x7f869c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:19.826 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86967fc700 1 -- 172.21.15.106:0/1817999894 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f869c0039c0 con 0x7f86a0107b20 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86967fc700 1 -- 172.21.15.106:0/1817999894 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f869c011e50 con 0x7f86a0107b20 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a019c290 con 0x7f86a0107b20 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.086+0000 7f86967fc700 1 -- 172.21.15.106:0/1817999894 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f869c01a5c0 con 0x7f86a0107b20 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.087+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a019ef80 con 0x7f86a0107b20 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.087+0000 7f86967fc700 1 -- 172.21.15.106:0/1817999894 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f869c01a8c0 con 0x7f86a0107b20 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.087+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a0069260 con 0x7f86a0107b20 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.087+0000 7f86967fc700 1 --2- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f868c038ed0 0x7f868c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.088+0000 7f86967fc700 1 -- 172.21.15.106:0/1817999894 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f869c04d260 con 0x7f86a0107b20 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.088+0000 7f86a4bc5700 1 --2- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f868c038ed0 0x7f868c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.088+0000 7f86a4bc5700 1 --2- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f868c038ed0 0x7f868c03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8690006fd0 tx=0x7f8690006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.092+0000 7f86967fc700 1 -- 172.21.15.106:0/1817999894 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f869c019c70 con 0x7f86a0107b20 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.289+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f86a01042e0 con 0x7f868c038ed0 2024-03-28T02:01:19.827 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.290+0000 7f86967fc700 1 -- 172.21.15.106:0/1817999894 <== mgr.14120 v2:172.21.15.106:6800/305853075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f86a01042e0 con 0x7f868c038ed0 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.292+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f868c038ed0 msgr2=0x7f868c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.292+0000 7f86a762a700 1 --2- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f868c038ed0 0x7f868c03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8690006fd0 tx=0x7f8690006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.292+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 msgr2=0x7f86a019bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.292+0000 7f86a762a700 1 --2- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a019bac0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f869c011670 tx=0x7f869c011b60 comp rx=0 tx=0).stop 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.292+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 shutdown_connections 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.292+0000 7f86a762a700 1 --2- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f868c038ed0 0x7f868c03b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.292+0000 7f86a762a700 1 --2- 172.21.15.106:0/1817999894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86a0107b20 0x7f86a019bac0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.292+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 >> 172.21.15.106:0/1817999894 conn(0x7f86a01030e0 msgr2=0x7f86a0103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.293+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 shutdown_connections 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:19.293+0000 7f86a762a700 1 -- 172.21.15.106:0/1817999894 wait complete. 2024-03-28T02:01:19.828 INFO:teuthology.orchestra.run.smithi106.stdout:Generating ssh key... 2024-03-28T02:01:20.406 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-mon[31965]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:20.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: Generating public/private rsa key pair. 2024-03-28T02:01:20.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: Your identification has been saved in /tmp/tmp943ddotj/key. 2024-03-28T02:01:20.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: Your public key has been saved in /tmp/tmp943ddotj/key.pub. 2024-03-28T02:01:20.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: The key fingerprint is: 2024-03-28T02:01:20.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: SHA256:8TAhNFz67bUXkkIc1LcEWlsUmmYo2RFgn/H9Zs8EX8k ceph-d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:01:20.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: The key's randomart image is: 2024-03-28T02:01:20.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: +---[RSA 3072]----+ 2024-03-28T02:01:20.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: | o+.=o*ooo+. | 2024-03-28T02:01:20.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: | .= * X.*+ .| 2024-03-28T02:01:20.824 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: | . * O BooE.| 2024-03-28T02:01:20.824 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: | . O o ..+.| 2024-03-28T02:01:20.824 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: | S + + . *| 2024-03-28T02:01:20.824 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: | . o o *.| 2024-03-28T02:01:20.824 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: | . . . o| 2024-03-28T02:01:20.824 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: | . | 2024-03-28T02:01:20.824 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: | | 2024-03-28T02:01:20.824 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: +----[SHA256]-----+ 2024-03-28T02:01:21.139 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.207+0000 7f7a5868b700 1 Processor -- start 2024-03-28T02:01:21.139 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.207+0000 7f7a5868b700 1 -- start start 2024-03-28T02:01:21.139 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a5868b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5007c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:21.139 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a5868b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5007c8c0 con 0x7f7a5007de40 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a56427700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5007c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a56427700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5007c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40534/0 (socket says 172.21.15.106:40534) 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a56427700 1 -- 172.21.15.106:0/4041321380 learned_addr learned my addr 172.21.15.106:0/4041321380 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a56427700 1 -- 172.21.15.106:0/4041321380 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5007ca40 con 0x7f7a5007de40 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a56427700 1 --2- 172.21.15.106:0/4041321380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5007c2f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a400098a0 tx=0x7f7a40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a449099add3044d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a55425700 1 -- 172.21.15.106:0/4041321380 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a40004160 con 0x7f7a5007de40 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a55425700 1 -- 172.21.15.106:0/4041321380 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a40004300 con 0x7f7a5007de40 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.208+0000 7f7a55425700 1 -- 172.21.15.106:0/4041321380 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a40011b50 con 0x7f7a5007de40 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.209+0000 7f7a5868b700 1 -- 172.21.15.106:0/4041321380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 msgr2=0x7f7a5007c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.209+0000 7f7a5868b700 1 --2- 172.21.15.106:0/4041321380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5007c2f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a400098a0 tx=0x7f7a40009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.209+0000 7f7a5868b700 1 -- 172.21.15.106:0/4041321380 shutdown_connections 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.209+0000 7f7a5868b700 1 --2- 172.21.15.106:0/4041321380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5007c2f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.209+0000 7f7a5868b700 1 -- 172.21.15.106:0/4041321380 >> 172.21.15.106:0/4041321380 conn(0x7f7a50103140 msgr2=0x7f7a50105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:21.140 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.209+0000 7f7a5868b700 1 -- 172.21.15.106:0/4041321380 shutdown_connections 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.209+0000 7f7a5868b700 1 -- 172.21.15.106:0/4041321380 wait complete. 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a5868b700 1 Processor -- start 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a5868b700 1 -- start start 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a5868b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5019dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a5868b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5010e110 con 0x7f7a5007de40 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a56427700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5019dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a56427700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5019dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40542/0 (socket says 172.21.15.106:40542) 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a56427700 1 -- 172.21.15.106:0/3368937115 learned_addr learned my addr 172.21.15.106:0/3368937115 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a56427700 1 -- 172.21.15.106:0/3368937115 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5019e240 con 0x7f7a5007de40 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a56427700 1 --2- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5019dd00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7a40011b60 tx=0x7f7a40011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a477fe700 1 -- 172.21.15.106:0/3368937115 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a40003970 con 0x7f7a5007de40 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a477fe700 1 -- 172.21.15.106:0/3368937115 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a40003b10 con 0x7f7a5007de40 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5019e4d0 con 0x7f7a5007de40 2024-03-28T02:01:21.141 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a477fe700 1 -- 172.21.15.106:0/3368937115 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a40024610 con 0x7f7a5007de40 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.210+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a501a11b0 con 0x7f7a5007de40 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.211+0000 7f7a477fe700 1 -- 172.21.15.106:0/3368937115 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7a4001a430 con 0x7f7a5007de40 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.211+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a50069260 con 0x7f7a5007de40 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.211+0000 7f7a477fe700 1 --2- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7a3c038f20 0x7f7a3c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.212+0000 7f7a477fe700 1 -- 172.21.15.106:0/3368937115 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7a40011e00 con 0x7f7a5007de40 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.212+0000 7f7a55c26700 1 --2- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7a3c038f20 0x7f7a3c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.212+0000 7f7a55c26700 1 --2- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7a3c038f20 0x7f7a3c03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c006fd0 tx=0x7f7a4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.216+0000 7f7a477fe700 1 -- 172.21.15.106:0/3368937115 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7a4000e260 con 0x7f7a5007de40 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.412+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a501043c0 con 0x7f7a3c038f20 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.521+0000 7f7a477fe700 1 -- 172.21.15.106:0/3368937115 <== mgr.14120 v2:172.21.15.106:6800/305853075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7a501043c0 con 0x7f7a3c038f20 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.523+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7a3c038f20 msgr2=0x7f7a3c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.523+0000 7f7a5868b700 1 --2- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7a3c038f20 0x7f7a3c03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c006fd0 tx=0x7f7a4c006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.523+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 msgr2=0x7f7a5019dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:21.142 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.523+0000 7f7a5868b700 1 --2- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5019dd00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7a40011b60 tx=0x7f7a40011c40 comp rx=0 tx=0).stop 2024-03-28T02:01:21.143 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.523+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 shutdown_connections 2024-03-28T02:01:21.143 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.523+0000 7f7a5868b700 1 --2- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7a3c038f20 0x7f7a3c03b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:21.143 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.523+0000 7f7a5868b700 1 --2- 172.21.15.106:0/3368937115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a5007de40 0x7f7a5019dd00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:21.143 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.523+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 >> 172.21.15.106:0/3368937115 conn(0x7f7a50103140 msgr2=0x7f7a50103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:21.143 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.524+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 shutdown_connections 2024-03-28T02:01:21.143 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:20.524+0000 7f7a5868b700 1 -- 172.21.15.106:0/3368937115 wait complete. 2024-03-28T02:01:21.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:21 smithi106 ceph-mon[31965]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:21.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:21 smithi106 ceph-mon[31965]: Generating ssh key... 2024-03-28T02:01:21.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:21 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:21.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:21 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:22.279 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdANol1AwH56UNCZECPEU81h8ybII4H7kKAwqzKyX8QkOXMukb0F2sh7judVQh5hOLMRyf3MY3pNIyf9EvzjWlEBFP3pKLasdbkbpWP13fcpDVfpsXrF+qBm4BbCn33fYbyulLKnfUFswrKNZxknY8HgTIcK5ZLl6FzZZzaG6iXLQy7GXZkBNI0wqYE/NvEdF84LE57RrycKjByzvUlh+E36BX8ag4h90/4Fsu9eLme6ZSmVYIS7zHdXxd5uBtBU+H39UPnqhAnuTAHmbhGGcDPR+Zr+p8rzDx52wzI0vxh/goT7Uv1vbDu2I6LmpyuM13V2z6KYjU0wId9BsIUEOKVDGt7AzFHM4CMmXIEyI3cFeTWuTM9IAii5Lb0gobPClR/HW64N1psp6xfwTu7zA7YAu6Jovjv4tMoE9Mnv3CkvwbMU70iyT3VpnekLlAjuwZNw4E21G4ic4P5LhhYwnZ/PIbTAL72VxckjlESPm0LPr0D9wctaDh+B0nXat2YOM= ceph-d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:01:22.279 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.505+0000 7f6ed5874700 1 Processor -- start 2024-03-28T02:01:22.279 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.505+0000 7f6ed5874700 1 -- start start 2024-03-28T02:01:22.279 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.505+0000 7f6ed5874700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed0106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:22.279 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.505+0000 7f6ed5874700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed0107400 con 0x7f6ed0106a10 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.505+0000 7f6eceffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed0106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.505+0000 7f6eceffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed0106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40544/0 (socket says 172.21.15.106:40544) 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.505+0000 7f6eceffd700 1 -- 172.21.15.106:0/2524764339 learned_addr learned my addr 172.21.15.106:0/2524764339 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.506+0000 7f6eceffd700 1 -- 172.21.15.106:0/2524764339 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0107c10 con 0x7f6ed0106a10 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.506+0000 7f6eceffd700 1 --2- 172.21.15.106:0/2524764339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed0106e30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00098a0 tx=0x7f6ec0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74320fad34ac2ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.506+0000 7f6ecdffb700 1 -- 172.21.15.106:0/2524764339 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec0004160 con 0x7f6ed0106a10 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.506+0000 7f6ecdffb700 1 -- 172.21.15.106:0/2524764339 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ec0004300 con 0x7f6ed0106a10 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.506+0000 7f6ecdffb700 1 -- 172.21.15.106:0/2524764339 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec0011b50 con 0x7f6ed0106a10 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.507+0000 7f6ed5874700 1 -- 172.21.15.106:0/2524764339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 msgr2=0x7f6ed0106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.507+0000 7f6ed5874700 1 --2- 172.21.15.106:0/2524764339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed0106e30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00098a0 tx=0x7f6ec0009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.507+0000 7f6ed5874700 1 -- 172.21.15.106:0/2524764339 shutdown_connections 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.507+0000 7f6ed5874700 1 --2- 172.21.15.106:0/2524764339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed0106e30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:22.280 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.507+0000 7f6ed5874700 1 -- 172.21.15.106:0/2524764339 >> 172.21.15.106:0/2524764339 conn(0x7f6ed0101ff0 msgr2=0x7f6ed0104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.507+0000 7f6ed5874700 1 -- 172.21.15.106:0/2524764339 shutdown_connections 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.507+0000 7f6ed5874700 1 -- 172.21.15.106:0/2524764339 wait complete. 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.507+0000 7f6ed5874700 1 Processor -- start 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.507+0000 7f6ed5874700 1 -- start start 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6ed5874700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed019bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6ed5874700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed010afc0 con 0x7f6ed0106a10 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6eceffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed019bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6eceffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed019bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40558/0 (socket says 172.21.15.106:40558) 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6eceffd700 1 -- 172.21.15.106:0/2750007164 learned_addr learned my addr 172.21.15.106:0/2750007164 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6eceffd700 1 -- 172.21.15.106:0/2750007164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed019c080 con 0x7f6ed0106a10 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6eceffd700 1 --2- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed019bb40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009fb0 tx=0x7f6ec0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6ed4872700 1 -- 172.21.15.106:0/2750007164 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec00038b0 con 0x7f6ed0106a10 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6ed4872700 1 -- 172.21.15.106:0/2750007164 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ec0011e50 con 0x7f6ed0106a10 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6ed4872700 1 -- 172.21.15.106:0/2750007164 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec0019550 con 0x7f6ed0106a10 2024-03-28T02:01:22.281 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed019c310 con 0x7f6ed0106a10 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.508+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed019f000 con 0x7f6ed0106a10 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.509+0000 7f6ed4872700 1 -- 172.21.15.106:0/2750007164 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f6ec0003a50 con 0x7f6ed0106a10 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.509+0000 7f6ed4872700 1 --2- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6eb8038f20 0x7f6eb803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.509+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed0069260 con 0x7f6ed0106a10 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.509+0000 7f6ed4872700 1 -- 172.21.15.106:0/2750007164 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6ec004d120 con 0x7f6ed0106a10 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.510+0000 7f6ece7fc700 1 --2- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6eb8038f20 0x7f6eb803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.510+0000 7f6ece7fc700 1 --2- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6eb8038f20 0x7f6eb803b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4006fd0 tx=0x7f6ec4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.514+0000 7f6ed4872700 1 -- 172.21.15.106:0/2750007164 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ec004cab0 con 0x7f6ed0106a10 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.711+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ed01032b0 con 0x7f6eb8038f20 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.711+0000 7f6ed4872700 1 -- 172.21.15.106:0/2750007164 <== mgr.14120 v2:172.21.15.106:6800/305853075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f6ed01032b0 con 0x7f6eb8038f20 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6eb8038f20 msgr2=0x7f6eb803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 --2- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6eb8038f20 0x7f6eb803b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4006fd0 tx=0x7f6ec4006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:22.282 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 msgr2=0x7f6ed019bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 --2- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed019bb40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009fb0 tx=0x7f6ec0011bf0 comp rx=0 tx=0).stop 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 shutdown_connections 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 --2- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6eb8038f20 0x7f6eb803b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 --2- 172.21.15.106:0/2750007164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ed0106a10 0x7f6ed019bb40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 >> 172.21.15.106:0/2750007164 conn(0x7f6ed0101ff0 msgr2=0x7f6ed0102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 shutdown_connections 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:21.714+0000 7f6ed5874700 1 -- 172.21.15.106:0/2750007164 wait complete. 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T02:01:22.283 INFO:teuthology.orchestra.run.smithi106.stdout:Adding host smithi106... 2024-03-28T02:01:22.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:22 smithi106 ceph-mon[31965]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:23.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:23 smithi106 ceph-mon[31965]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi106", "addr": "172.21.15.106", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:24.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:24 smithi106 ceph-mon[31965]: Deploying cephadm binary to smithi106 2024-03-28T02:01:27.790 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Added host 'smithi106' with addr '172.21.15.106' 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.706+0000 7fa7388f8700 1 Processor -- start 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.706+0000 7fa7388f8700 1 -- start start 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.706+0000 7fa7388f8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734106e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.706+0000 7fa7388f8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa734107420 con 0x7fa734106a30 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.707+0000 7fa73259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734106e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.707+0000 7fa73259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734106e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40566/0 (socket says 172.21.15.106:40566) 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.707+0000 7fa73259c700 1 -- 172.21.15.106:0/3996515324 learned_addr learned my addr 172.21.15.106:0/3996515324 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.707+0000 7fa73259c700 1 -- 172.21.15.106:0/3996515324 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734107c30 con 0x7fa734106a30 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.707+0000 7fa73259c700 1 --2- 172.21.15.106:0/3996515324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734106e50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009e80 tx=0x7fa72c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5e51e70d2e35ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.707+0000 7fa73159a700 1 -- 172.21.15.106:0/3996515324 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa72c017070 con 0x7fa734106a30 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.707+0000 7fa73159a700 1 -- 172.21.15.106:0/3996515324 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa72c00b8c0 con 0x7fa734106a30 2024-03-28T02:01:27.791 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.707+0000 7fa73159a700 1 -- 172.21.15.106:0/3996515324 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa72c0045c0 con 0x7fa734106a30 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.708+0000 7fa7388f8700 1 -- 172.21.15.106:0/3996515324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 msgr2=0x7fa734106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.708+0000 7fa7388f8700 1 --2- 172.21.15.106:0/3996515324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734106e50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009e80 tx=0x7fa72c00b330 comp rx=0 tx=0).stop 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.708+0000 7fa7388f8700 1 -- 172.21.15.106:0/3996515324 shutdown_connections 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.708+0000 7fa7388f8700 1 --2- 172.21.15.106:0/3996515324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734106e50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.708+0000 7fa7388f8700 1 -- 172.21.15.106:0/3996515324 >> 172.21.15.106:0/3996515324 conn(0x7fa734101ff0 msgr2=0x7fa734104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.708+0000 7fa7388f8700 1 -- 172.21.15.106:0/3996515324 shutdown_connections 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.708+0000 7fa7388f8700 1 -- 172.21.15.106:0/3996515324 wait complete. 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa7388f8700 1 Processor -- start 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa7388f8700 1 -- start start 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa7388f8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa7388f8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73410afe0 con 0x7fa734106a30 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa73259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa73259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40580/0 (socket says 172.21.15.106:40580) 2024-03-28T02:01:27.792 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa73259c700 1 -- 172.21.15.106:0/1582156572 learned_addr learned my addr 172.21.15.106:0/1582156572 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa73259c700 1 -- 172.21.15.106:0/1582156572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734190310 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa73259c700 1 --2- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734191dc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009530 tx=0x7fa72c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.709+0000 7fa72b7fe700 1 -- 172.21.15.106:0/1582156572 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa72c020020 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.710+0000 7fa72b7fe700 1 -- 172.21.15.106:0/1582156572 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa72c003b50 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.710+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7341905a0 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.710+0000 7fa72b7fe700 1 -- 172.21.15.106:0/1582156572 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa72c017040 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.710+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa734190ae0 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.710+0000 7fa72b7fe700 1 -- 172.21.15.106:0/1582156572 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa72c023020 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.710+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7340637c0 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.711+0000 7fa72b7fe700 1 --2- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fa720038f20 0x7fa72003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.711+0000 7fa72b7fe700 1 -- 172.21.15.106:0/1582156572 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa72c00f050 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.711+0000 7fa731d9b700 1 --2- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fa720038f20 0x7fa72003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.711+0000 7fa731d9b700 1 --2- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fa720038f20 0x7fa72003b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa71c006fd0 tx=0x7fa71c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.715+0000 7fa72b7fe700 1 -- 172.21.15.106:0/1582156572 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa72c02a020 con 0x7fa734106a30 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:22.916+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi106", "addr": "172.21.15.106", "target": ["mon-mgr", ""]}) v1 -- 0x7fa734191100 con 0x7fa720038f20 2024-03-28T02:01:27.793 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.188+0000 7fa72b7fe700 1 -- 172.21.15.106:0/1582156572 <== mgr.14120 v2:172.21.15.106:6800/305853075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa734191100 con 0x7fa720038f20 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fa720038f20 msgr2=0x7fa72003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 --2- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fa720038f20 0x7fa72003b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa71c006fd0 tx=0x7fa71c006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 msgr2=0x7fa734191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 --2- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734191dc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009530 tx=0x7fa72c00bd50 comp rx=0 tx=0).stop 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 shutdown_connections 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 --2- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fa720038f20 0x7fa72003b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 --2- 172.21.15.106:0/1582156572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734106a30 0x7fa734191dc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 >> 172.21.15.106:0/1582156572 conn(0x7fa734101ff0 msgr2=0x7fa734102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 shutdown_connections 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:27.191+0000 7fa7388f8700 1 -- 172.21.15.106:0/1582156572 wait complete. 2024-03-28T02:01:27.794 INFO:teuthology.orchestra.run.smithi106.stdout:Deploying unmanaged mon service... 2024-03-28T02:01:28.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:28 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:28.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:28 smithi106 ceph-mon[31965]: Added host smithi106 2024-03-28T02:01:28.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:28 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:29.374 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T02:01:29.374 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.323+0000 7facaee97700 1 Processor -- start 2024-03-28T02:01:29.374 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.323+0000 7facaee97700 1 -- start start 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.323+0000 7facaee97700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca8108410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.323+0000 7facaee97700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca81089e0 con 0x7faca8107ff0 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.323+0000 7facacc33700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca8108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.323+0000 7facacc33700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca8108410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54164/0 (socket says 172.21.15.106:54164) 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.323+0000 7facacc33700 1 -- 172.21.15.106:0/3574030679 learned_addr learned my addr 172.21.15.106:0/3574030679 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.323+0000 7facacc33700 1 -- 172.21.15.106:0/3574030679 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8106ca0 con 0x7faca8107ff0 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.323+0000 7facacc33700 1 --2- 172.21.15.106:0/3574030679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca8108410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0098a0 tx=0x7fac9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05f97a6663fd254 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7faca77fe700 1 -- 172.21.15.106:0/3574030679 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac9c004160 con 0x7faca8107ff0 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7faca77fe700 1 -- 172.21.15.106:0/3574030679 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac9c004300 con 0x7faca8107ff0 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7faca77fe700 1 -- 172.21.15.106:0/3574030679 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac9c011b50 con 0x7faca8107ff0 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7facaee97700 1 -- 172.21.15.106:0/3574030679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 msgr2=0x7faca8108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7facaee97700 1 --2- 172.21.15.106:0/3574030679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca8108410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0098a0 tx=0x7fac9c009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:29.375 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7facaee97700 1 -- 172.21.15.106:0/3574030679 shutdown_connections 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7facaee97700 1 --2- 172.21.15.106:0/3574030679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca8108410 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7facaee97700 1 -- 172.21.15.106:0/3574030679 >> 172.21.15.106:0/3574030679 conn(0x7faca8101750 msgr2=0x7faca8103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7facaee97700 1 -- 172.21.15.106:0/3574030679 shutdown_connections 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.324+0000 7facaee97700 1 -- 172.21.15.106:0/3574030679 wait complete. 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.325+0000 7facaee97700 1 Processor -- start 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.325+0000 7facaee97700 1 -- start start 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.325+0000 7facaee97700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca819e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.325+0000 7facaee97700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca810b060 con 0x7faca8107ff0 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.325+0000 7facacc33700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca819e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.325+0000 7facacc33700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca819e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54178/0 (socket says 172.21.15.106:54178) 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.325+0000 7facacc33700 1 -- 172.21.15.106:0/1446254026 learned_addr learned my addr 172.21.15.106:0/1446254026 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.325+0000 7facacc33700 1 -- 172.21.15.106:0/1446254026 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca81067f0 con 0x7faca8107ff0 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.325+0000 7facacc33700 1 --2- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca819e8a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009fb0 tx=0x7fac9c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.326+0000 7faca5ffb700 1 -- 172.21.15.106:0/1446254026 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac9c0038b0 con 0x7faca8107ff0 2024-03-28T02:01:29.376 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.326+0000 7faca5ffb700 1 -- 172.21.15.106:0/1446254026 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac9c011e50 con 0x7faca8107ff0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.326+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca8106a80 con 0x7faca8107ff0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.326+0000 7faca5ffb700 1 -- 172.21.15.106:0/1446254026 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac9c019550 con 0x7faca8107ff0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.326+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca819f030 con 0x7faca8107ff0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.327+0000 7faca5ffb700 1 -- 172.21.15.106:0/1446254026 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fac9c003a50 con 0x7faca8107ff0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.327+0000 7faca5ffb700 1 --2- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fac98038ed0 0x7fac9803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.327+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca80637c0 con 0x7faca8107ff0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.327+0000 7faca5ffb700 1 -- 172.21.15.106:0/1446254026 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fac9c04d170 con 0x7faca8107ff0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.327+0000 7faca7fff700 1 --2- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fac98038ed0 0x7fac9803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.327+0000 7faca7fff700 1 --2- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fac98038ed0 0x7fac9803b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fac94006fd0 tx=0x7fac94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.332+0000 7faca5ffb700 1 -- 172.21.15.106:0/1446254026 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fac9c04cb00 con 0x7faca8107ff0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.550+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7faca8102a60 con 0x7fac98038ed0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.557+0000 7faca5ffb700 1 -- 172.21.15.106:0/1446254026 <== mgr.14120 v2:172.21.15.106:6800/305853075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7faca8102a60 con 0x7fac98038ed0 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fac98038ed0 msgr2=0x7fac9803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 --2- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fac98038ed0 0x7fac9803b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fac94006fd0 tx=0x7fac94006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:29.378 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 msgr2=0x7faca819e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:29.378 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 --2- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca819e8a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009fb0 tx=0x7fac9c011bf0 comp rx=0 tx=0).stop 2024-03-28T02:01:29.378 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 shutdown_connections 2024-03-28T02:01:29.378 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 --2- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fac98038ed0 0x7fac9803b390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:29.378 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 --2- 172.21.15.106:0/1446254026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faca8107ff0 0x7faca819e8a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:29.378 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 >> 172.21.15.106:0/1446254026 conn(0x7faca8101750 msgr2=0x7faca8102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:29.378 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 shutdown_connections 2024-03-28T02:01:29.378 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:28.560+0000 7facaee97700 1 -- 172.21.15.106:0/1446254026 wait complete. 2024-03-28T02:01:29.378 INFO:teuthology.orchestra.run.smithi106.stdout:Deploying unmanaged mgr service... 2024-03-28T02:01:29.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:29 smithi106 ceph-mon[31965]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:29.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:29 smithi106 ceph-mon[31965]: Saving service mon spec with placement count:5 2024-03-28T02:01:29.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:29 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:30.702 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T02:01:30.702 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.931+0000 7fb24b59e700 1 Processor -- start 2024-03-28T02:01:30.702 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.931+0000 7fb24b59e700 1 -- start start 2024-03-28T02:01:30.702 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.931+0000 7fb24b59e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c108240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:30.702 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.931+0000 7fb24b59e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c108810 con 0x7fb24c107e20 2024-03-28T02:01:30.702 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.931+0000 7fb24a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.931+0000 7fb24a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54186/0 (socket says 172.21.15.106:54186) 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.931+0000 7fb24a59c700 1 -- 172.21.15.106:0/3096939721 learned_addr learned my addr 172.21.15.106:0/3096939721 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.932+0000 7fb24a59c700 1 -- 172.21.15.106:0/3096939721 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c106a30 con 0x7fb24c107e20 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.932+0000 7fb24a59c700 1 --2- 172.21.15.106:0/3096939721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c108240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00bb40 tx=0x7fb23c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d914a99852ccee9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.932+0000 7fb249d9b700 1 -- 172.21.15.106:0/3096939721 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb23c00dbe0 con 0x7fb24c107e20 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.932+0000 7fb249d9b700 1 -- 172.21.15.106:0/3096939721 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb23c0044f0 con 0x7fb24c107e20 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.932+0000 7fb249d9b700 1 -- 172.21.15.106:0/3096939721 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb23c010470 con 0x7fb24c107e20 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.933+0000 7fb24b59e700 1 -- 172.21.15.106:0/3096939721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 msgr2=0x7fb24c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.933+0000 7fb24b59e700 1 --2- 172.21.15.106:0/3096939721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c108240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00bb40 tx=0x7fb23c00be50 comp rx=0 tx=0).stop 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.933+0000 7fb24b59e700 1 -- 172.21.15.106:0/3096939721 shutdown_connections 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.933+0000 7fb24b59e700 1 --2- 172.21.15.106:0/3096939721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c108240 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:30.703 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.933+0000 7fb24b59e700 1 -- 172.21.15.106:0/3096939721 >> 172.21.15.106:0/3096939721 conn(0x7fb24c101580 msgr2=0x7fb24c1039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.933+0000 7fb24b59e700 1 -- 172.21.15.106:0/3096939721 shutdown_connections 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.933+0000 7fb24b59e700 1 -- 172.21.15.106:0/3096939721 wait complete. 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.933+0000 7fb24b59e700 1 Processor -- start 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.934+0000 7fb24b59e700 1 -- start start 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.934+0000 7fb24b59e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.934+0000 7fb24b59e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c105920 con 0x7fb24c107e20 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.934+0000 7fb24a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.934+0000 7fb24a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54194/0 (socket says 172.21.15.106:54194) 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.934+0000 7fb24a59c700 1 -- 172.21.15.106:0/1320951897 learned_addr learned my addr 172.21.15.106:0/1320951897 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.934+0000 7fb24a59c700 1 -- 172.21.15.106:0/1320951897 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c199c70 con 0x7fb24c107e20 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.934+0000 7fb24a59c700 1 --2- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c199730 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00b340 tx=0x7fb23c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.935+0000 7fb243fff700 1 -- 172.21.15.106:0/1320951897 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb23c027070 con 0x7fb24c107e20 2024-03-28T02:01:30.704 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.935+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24c199f00 con 0x7fb24c107e20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.935+0000 7fb243fff700 1 -- 172.21.15.106:0/1320951897 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb23c010b40 con 0x7fb24c107e20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.935+0000 7fb243fff700 1 -- 172.21.15.106:0/1320951897 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb23c022b00 con 0x7fb24c107e20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.935+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24c19cbf0 con 0x7fb24c107e20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.935+0000 7fb243fff700 1 -- 172.21.15.106:0/1320951897 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb23c02a070 con 0x7fb24c107e20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.936+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24c069260 con 0x7fb24c107e20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.936+0000 7fb243fff700 1 --2- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fb238038f20 0x7fb23803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.936+0000 7fb243fff700 1 -- 172.21.15.106:0/1320951897 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb23c01f070 con 0x7fb24c107e20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.936+0000 7fb241d9b700 1 --2- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fb238038f20 0x7fb23803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.937+0000 7fb241d9b700 1 --2- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fb238038f20 0x7fb23803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb234006fd0 tx=0x7fb234006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:29.941+0000 7fb243fff700 1 -- 172.21.15.106:0/1320951897 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb23c015070 con 0x7fb24c107e20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.152+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb24c0008d0 con 0x7fb238038f20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.156+0000 7fb243fff700 1 -- 172.21.15.106:0/1320951897 <== mgr.14120 v2:172.21.15.106:6800/305853075 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb24c0008d0 con 0x7fb238038f20 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fb238038f20 msgr2=0x7fb23803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 --2- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fb238038f20 0x7fb23803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb234006fd0 tx=0x7fb234006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 msgr2=0x7fb24c199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:30.705 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 --2- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c199730 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00b340 tx=0x7fb23c00dd90 comp rx=0 tx=0).stop 2024-03-28T02:01:30.706 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 shutdown_connections 2024-03-28T02:01:30.706 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 --2- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fb238038f20 0x7fb23803b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:30.706 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 --2- 172.21.15.106:0/1320951897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c107e20 0x7fb24c199730 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:30.706 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 >> 172.21.15.106:0/1320951897 conn(0x7fb24c101580 msgr2=0x7fb24c190210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:30.706 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 shutdown_connections 2024-03-28T02:01:30.706 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:30.158+0000 7fb24b59e700 1 -- 172.21.15.106:0/1320951897 wait complete. 2024-03-28T02:01:31.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:31 smithi106 ceph-mon[31965]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:31.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:31 smithi106 ceph-mon[31965]: Saving service mgr spec with placement count:2 2024-03-28T02:01:31.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:31 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:31.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:31 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:31.949 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.154+0000 7f6b23791700 1 Processor -- start 2024-03-28T02:01:31.949 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.154+0000 7f6b23791700 1 -- start start 2024-03-28T02:01:31.949 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.154+0000 7f6b23791700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c107f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:31.949 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.154+0000 7f6b23791700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c108510 con 0x7f6b1c107b20 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.154+0000 7f6b2152d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c107f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.154+0000 7f6b2152d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c107f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54202/0 (socket says 172.21.15.106:54202) 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.154+0000 7f6b2152d700 1 -- 172.21.15.106:0/2335804359 learned_addr learned my addr 172.21.15.106:0/2335804359 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.155+0000 7f6b2152d700 1 -- 172.21.15.106:0/2335804359 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c108d20 con 0x7f6b1c107b20 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.155+0000 7f6b2152d700 1 --2- 172.21.15.106:0/2335804359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c107f40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6b180098a0 tx=0x7f6b18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cfe43ec5e64957 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.159+0000 7f6b13fff700 1 -- 172.21.15.106:0/2335804359 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b18004160 con 0x7f6b1c107b20 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.159+0000 7f6b13fff700 1 -- 172.21.15.106:0/2335804359 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b18004300 con 0x7f6b1c107b20 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.159+0000 7f6b13fff700 1 -- 172.21.15.106:0/2335804359 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b18011b20 con 0x7f6b1c107b20 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.159+0000 7f6b23791700 1 -- 172.21.15.106:0/2335804359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 msgr2=0x7f6b1c107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.159+0000 7f6b23791700 1 --2- 172.21.15.106:0/2335804359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c107f40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6b180098a0 tx=0x7f6b18009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.159+0000 7f6b23791700 1 -- 172.21.15.106:0/2335804359 shutdown_connections 2024-03-28T02:01:31.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.159+0000 7f6b23791700 1 --2- 172.21.15.106:0/2335804359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c107f40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.159+0000 7f6b23791700 1 -- 172.21.15.106:0/2335804359 >> 172.21.15.106:0/2335804359 conn(0x7f6b1c1030e0 msgr2=0x7f6b1c105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.160+0000 7f6b23791700 1 -- 172.21.15.106:0/2335804359 shutdown_connections 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.160+0000 7f6b23791700 1 -- 172.21.15.106:0/2335804359 wait complete. 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.160+0000 7f6b23791700 1 Processor -- start 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.160+0000 7f6b23791700 1 -- start start 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.160+0000 7f6b23791700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.160+0000 7f6b23791700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c10c0d0 con 0x7f6b1c107b20 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.160+0000 7f6b2152d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.160+0000 7f6b2152d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c19ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54216/0 (socket says 172.21.15.106:54216) 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.161+0000 7f6b2152d700 1 -- 172.21.15.106:0/1526227023 learned_addr learned my addr 172.21.15.106:0/1526227023 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.161+0000 7f6b2152d700 1 -- 172.21.15.106:0/1526227023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c19bf90 con 0x7f6b1c107b20 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.161+0000 7f6b2152d700 1 --2- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c19ba50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6b180117e0 tx=0x7f6b18011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.161+0000 7f6b127fc700 1 -- 172.21.15.106:0/1526227023 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b18003bf0 con 0x7f6b1c107b20 2024-03-28T02:01:31.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.161+0000 7f6b127fc700 1 -- 172.21.15.106:0/1526227023 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b1801a910 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.161+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1c19c220 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.161+0000 7f6b127fc700 1 -- 172.21.15.106:0/1526227023 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b18024970 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.161+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1c19c680 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.162+0000 7f6b127fc700 1 -- 172.21.15.106:0/1526227023 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f6b18022020 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.162+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1c069260 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.162+0000 7f6b127fc700 1 --2- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6b08038f70 0x7f6b0803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.162+0000 7f6b127fc700 1 -- 172.21.15.106:0/1526227023 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6b1804d290 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.162+0000 7f6b20d2c700 1 --2- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6b08038f70 0x7f6b0803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.163+0000 7f6b20d2c700 1 --2- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6b08038f70 0x7f6b0803b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c006fd0 tx=0x7f6b0c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.167+0000 7f6b127fc700 1 -- 172.21.15.106:0/1526227023 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b1802d910 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.374+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f6b1c19ef10 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.379+0000 7f6b127fc700 1 -- 172.21.15.106:0/1526227023 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f6b1802d3f0 con 0x7f6b1c107b20 2024-03-28T02:01:31.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6b08038f70 msgr2=0x7f6b0803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:31.953 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 --2- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6b08038f70 0x7f6b0803b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c006fd0 tx=0x7f6b0c006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:31.953 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 msgr2=0x7f6b1c19ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:31.953 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 --2- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c19ba50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6b180117e0 tx=0x7f6b18011ba0 comp rx=0 tx=0).stop 2024-03-28T02:01:31.953 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 shutdown_connections 2024-03-28T02:01:31.953 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 --2- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f6b08038f70 0x7f6b0803b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:31.953 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 --2- 172.21.15.106:0/1526227023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b1c107b20 0x7f6b1c19ba50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:31.953 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 >> 172.21.15.106:0/1526227023 conn(0x7f6b1c1030e0 msgr2=0x7f6b1c103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:31.953 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 shutdown_connections 2024-03-28T02:01:31.953 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:31.381+0000 7f6b23791700 1 -- 172.21.15.106:0/1526227023 wait complete. 2024-03-28T02:01:32.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:32 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:32.550 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:32 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1526227023' entity='client.admin' 2024-03-28T02:01:33.693 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.313+0000 7faac0136700 1 Processor -- start 2024-03-28T02:01:33.693 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.313+0000 7faac0136700 1 -- start start 2024-03-28T02:01:33.693 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.314+0000 7faac0136700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab8106810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.314+0000 7faac0136700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab807be00 con 0x7faab81063f0 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.314+0000 7faabded2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab8106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.314+0000 7faabded2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab8106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54222/0 (socket says 172.21.15.106:54222) 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.314+0000 7faabded2700 1 -- 172.21.15.106:0/23865082 learned_addr learned my addr 172.21.15.106:0/23865082 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.314+0000 7faabded2700 1 -- 172.21.15.106:0/23865082 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab8106d50 con 0x7faab81063f0 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.314+0000 7faabded2700 1 --2- 172.21.15.106:0/23865082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab8106810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faab40098a0 tx=0x7faab4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3dc77862ee7da99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.315+0000 7faabced0700 1 -- 172.21.15.106:0/23865082 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faab4004160 con 0x7faab81063f0 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.315+0000 7faabced0700 1 -- 172.21.15.106:0/23865082 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faab4004300 con 0x7faab81063f0 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.315+0000 7faabced0700 1 -- 172.21.15.106:0/23865082 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faab4011b20 con 0x7faab81063f0 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.315+0000 7faac0136700 1 -- 172.21.15.106:0/23865082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 msgr2=0x7faab8106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.315+0000 7faac0136700 1 --2- 172.21.15.106:0/23865082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab8106810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faab40098a0 tx=0x7faab4009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.315+0000 7faac0136700 1 -- 172.21.15.106:0/23865082 shutdown_connections 2024-03-28T02:01:33.694 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.316+0000 7faac0136700 1 --2- 172.21.15.106:0/23865082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab8106810 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.316+0000 7faac0136700 1 -- 172.21.15.106:0/23865082 >> 172.21.15.106:0/23865082 conn(0x7faab8101ff0 msgr2=0x7faab8104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.316+0000 7faac0136700 1 -- 172.21.15.106:0/23865082 shutdown_connections 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.316+0000 7faac0136700 1 -- 172.21.15.106:0/23865082 wait complete. 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.316+0000 7faac0136700 1 Processor -- start 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.316+0000 7faac0136700 1 -- start start 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.316+0000 7faac0136700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.316+0000 7faac0136700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab8107e20 con 0x7faab81063f0 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faabded2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faabded2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54234/0 (socket says 172.21.15.106:54234) 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faabded2700 1 -- 172.21.15.106:0/2456546714 learned_addr learned my addr 172.21.15.106:0/2456546714 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faabded2700 1 -- 172.21.15.106:0/2456546714 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab819e350 con 0x7faab81063f0 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faabded2700 1 --2- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab819de10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faab40117e0 tx=0x7faab4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faaaeffd700 1 -- 172.21.15.106:0/2456546714 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faab4003ba0 con 0x7faab81063f0 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faaaeffd700 1 -- 172.21.15.106:0/2456546714 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faab401a910 con 0x7faab81063f0 2024-03-28T02:01:33.695 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab819e5e0 con 0x7faab81063f0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faaaeffd700 1 -- 172.21.15.106:0/2456546714 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faab40248f0 con 0x7faab81063f0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.317+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab819efc0 con 0x7faab81063f0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.318+0000 7faaaeffd700 1 -- 172.21.15.106:0/2456546714 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7faab4022020 con 0x7faab81063f0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.318+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab80637c0 con 0x7faab81063f0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.319+0000 7faaaeffd700 1 --2- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7faaa4038f70 0x7faaa403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.319+0000 7faaaeffd700 1 -- 172.21.15.106:0/2456546714 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faab401aac0 con 0x7faab81063f0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.319+0000 7faabd6d1700 1 --2- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7faaa4038f70 0x7faaa403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.319+0000 7faabd6d1700 1 --2- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7faaa4038f70 0x7faaa403b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faaa8006fd0 tx=0x7faaa8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.324+0000 7faaaeffd700 1 -- 172.21.15.106:0/2456546714 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faab40172b0 con 0x7faab81063f0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.555+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7faab819fc90 con 0x7faab81063f0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.558+0000 7faaaeffd700 1 -- 172.21.15.106:0/2456546714 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7faab400b5a0 con 0x7faab81063f0 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7faaa4038f70 msgr2=0x7faaa403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:33.696 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 --2- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7faaa4038f70 0x7faaa403b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faaa8006fd0 tx=0x7faaa8006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:33.697 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 msgr2=0x7faab819de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:33.697 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 --2- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab819de10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faab40117e0 tx=0x7faab4011ba0 comp rx=0 tx=0).stop 2024-03-28T02:01:33.697 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 shutdown_connections 2024-03-28T02:01:33.697 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 --2- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7faaa4038f70 0x7faaa403b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:33.697 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 --2- 172.21.15.106:0/2456546714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faab81063f0 0x7faab819de10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:33.697 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 >> 172.21.15.106:0/2456546714 conn(0x7faab8101ff0 msgr2=0x7faab81948b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:33.697 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 shutdown_connections 2024-03-28T02:01:33.697 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:32.560+0000 7faac0136700 1 -- 172.21.15.106:0/2456546714 wait complete. 2024-03-28T02:01:33.697 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling the dashboard module... 2024-03-28T02:01:33.809 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:33 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2456546714' entity='client.admin' 2024-03-28T02:01:33.809 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:33 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:35.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:35 smithi106 ceph-mon[31965]: from='mgr.14120 172.21.15.106:0/3220231394' entity='mgr.x' 2024-03-28T02:01:35.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:35 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3370100830' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T02:01:35.584 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:35 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: ignoring --setuser ceph since I am not root 2024-03-28T02:01:35.584 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:35 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: ignoring --setgroup ceph since I am not root 2024-03-28T02:01:36.050 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.224+0000 7efd8eb62700 1 Processor -- start 2024-03-28T02:01:36.055 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.225+0000 7efd8eb62700 1 -- start start 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.225+0000 7efd8eb62700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8807c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.225+0000 7efd8eb62700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8807c8a0 con 0x7efd8807de70 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.225+0000 7efd8c8fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8807c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.225+0000 7efd8c8fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8807c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54242/0 (socket says 172.21.15.106:54242) 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.225+0000 7efd8c8fe700 1 -- 172.21.15.106:0/2915296584 learned_addr learned my addr 172.21.15.106:0/2915296584 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.225+0000 7efd8c8fe700 1 -- 172.21.15.106:0/2915296584 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8807ca20 con 0x7efd8807de70 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.226+0000 7efd8c8fe700 1 --2- 172.21.15.106:0/2915296584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8807c2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0098a0 tx=0x7efd7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a8e5c50667e971 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.226+0000 7efd877fe700 1 -- 172.21.15.106:0/2915296584 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd7c004160 con 0x7efd8807de70 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.226+0000 7efd877fe700 1 -- 172.21.15.106:0/2915296584 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd7c004300 con 0x7efd8807de70 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.226+0000 7efd877fe700 1 -- 172.21.15.106:0/2915296584 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd7c011b20 con 0x7efd8807de70 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.226+0000 7efd8eb62700 1 -- 172.21.15.106:0/2915296584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 msgr2=0x7efd8807c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.226+0000 7efd8eb62700 1 --2- 172.21.15.106:0/2915296584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8807c2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0098a0 tx=0x7efd7c009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.226+0000 7efd8eb62700 1 -- 172.21.15.106:0/2915296584 shutdown_connections 2024-03-28T02:01:36.056 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.226+0000 7efd8eb62700 1 --2- 172.21.15.106:0/2915296584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8807c2d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.226+0000 7efd8eb62700 1 -- 172.21.15.106:0/2915296584 >> 172.21.15.106:0/2915296584 conn(0x7efd88103190 msgr2=0x7efd881055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.227+0000 7efd8eb62700 1 -- 172.21.15.106:0/2915296584 shutdown_connections 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.227+0000 7efd8eb62700 1 -- 172.21.15.106:0/2915296584 wait complete. 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.227+0000 7efd8eb62700 1 Processor -- start 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.227+0000 7efd8eb62700 1 -- start start 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.227+0000 7efd8eb62700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.227+0000 7efd8eb62700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8810e150 con 0x7efd8807de70 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.227+0000 7efd8c8fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.227+0000 7efd8c8fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8819de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54252/0 (socket says 172.21.15.106:54252) 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.227+0000 7efd8c8fe700 1 -- 172.21.15.106:0/3370100830 learned_addr learned my addr 172.21.15.106:0/3370100830 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.228+0000 7efd8c8fe700 1 -- 172.21.15.106:0/3370100830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8819e380 con 0x7efd8807de70 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.228+0000 7efd8c8fe700 1 --2- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8819de40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd7c000c00 tx=0x7efd7c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.228+0000 7efd85ffb700 1 -- 172.21.15.106:0/3370100830 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd7c003b60 con 0x7efd8807de70 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.228+0000 7efd85ffb700 1 -- 172.21.15.106:0/3370100830 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd7c01a910 con 0x7efd8807de70 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.228+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8819e610 con 0x7efd8807de70 2024-03-28T02:01:36.057 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.228+0000 7efd85ffb700 1 -- 172.21.15.106:0/3370100830 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd7c024930 con 0x7efd8807de70 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.228+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8819eff0 con 0x7efd8807de70 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.229+0000 7efd85ffb700 1 -- 172.21.15.106:0/3370100830 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7efd7c022020 con 0x7efd8807de70 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.229+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd880637c0 con 0x7efd8807de70 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.229+0000 7efd85ffb700 1 --2- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7efd78038f20 0x7efd7803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.229+0000 7efd85ffb700 1 -- 172.21.15.106:0/3370100830 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efd7c04d1f0 con 0x7efd8807de70 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.230+0000 7efd87fff700 1 --2- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7efd78038f20 0x7efd7803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.230+0000 7efd87fff700 1 --2- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7efd78038f20 0x7efd7803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efd74006fd0 tx=0x7efd74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.235+0000 7efd85ffb700 1 -- 172.21.15.106:0/3370100830 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efd7c02d3f0 con 0x7efd8807de70 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:34.496+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7efd8819f4e0 con 0x7efd8807de70 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.454+0000 7efd85ffb700 1 -- 172.21.15.106:0/3370100830 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7efd7c00b7b0 con 0x7efd8807de70 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.456+0000 7efd85ffb700 1 -- 172.21.15.106:0/3370100830 <== mon.0 v2:172.21.15.106:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7efd7c0193c0 con 0x7efd8807de70 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.457+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7efd78038f20 msgr2=0x7efd7803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.457+0000 7efd8eb62700 1 --2- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7efd78038f20 0x7efd7803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efd74006fd0 tx=0x7efd74006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.457+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 msgr2=0x7efd8819de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.457+0000 7efd8eb62700 1 --2- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8819de40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd7c000c00 tx=0x7efd7c011c10 comp rx=0 tx=0).stop 2024-03-28T02:01:36.058 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.458+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 shutdown_connections 2024-03-28T02:01:36.059 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.458+0000 7efd8eb62700 1 --2- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7efd78038f20 0x7efd7803b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:36.059 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.458+0000 7efd8eb62700 1 --2- 172.21.15.106:0/3370100830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd8807de70 0x7efd8819de40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:36.059 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.458+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 >> 172.21.15.106:0/3370100830 conn(0x7efd88103190 msgr2=0x7efd88103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:36.059 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.458+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 shutdown_connections 2024-03-28T02:01:36.059 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:35.458+0000 7efd8eb62700 1 -- 172.21.15.106:0/3370100830 wait complete. 2024-03-28T02:01:36.268 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:36 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:36.020+0000 7efc5b7bd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T02:01:36.268 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:36 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:36.120+0000 7efc5b7bd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T02:01:36.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:36 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3370100830' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T02:01:36.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:36 smithi106 ceph-mon[31965]: mgrmap e9: x(active, since 19s) 2024-03-28T02:01:36.919 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:36 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:36.775+0000 7efc5b7bd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T02:01:36.919 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:36 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:36.867+0000 7efc5b7bd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T02:01:37.181 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:37.009+0000 7efc5b7bd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T02:01:37.181 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:37.103+0000 7efc5b7bd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T02:01:37.181 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:37.186+0000 7efc5b7bd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T02:01:37.481 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-28T02:01:37.481 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T02:01:37.481 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T02:01:37.481 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T02:01:37.481 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T02:01:37.481 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-28T02:01:37.481 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.462+0000 7f7e240c7700 1 Processor -- start 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.462+0000 7f7e240c7700 1 -- start start 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.462+0000 7f7e240c7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c10a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.462+0000 7f7e240c7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c07be20 con 0x7f7e1c1085c0 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.462+0000 7f7e21e63700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c10a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.462+0000 7f7e21e63700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c10a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54292/0 (socket says 172.21.15.106:54292) 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.462+0000 7f7e21e63700 1 -- 172.21.15.106:0/1272786659 learned_addr learned my addr 172.21.15.106:0/1272786659 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.463+0000 7f7e21e63700 1 -- 172.21.15.106:0/1272786659 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c10aef0 con 0x7f7e1c1085c0 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.463+0000 7f7e21e63700 1 --2- 172.21.15.106:0/1272786659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c10a9b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e180098a0 tx=0x7f7e18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21b3e62eccf7580 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.463+0000 7f7e20e61700 1 -- 172.21.15.106:0/1272786659 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e18004160 con 0x7f7e1c1085c0 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.463+0000 7f7e20e61700 1 -- 172.21.15.106:0/1272786659 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e18004300 con 0x7f7e1c1085c0 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.463+0000 7f7e20e61700 1 -- 172.21.15.106:0/1272786659 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e18011b20 con 0x7f7e1c1085c0 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.464+0000 7f7e240c7700 1 -- 172.21.15.106:0/1272786659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 msgr2=0x7f7e1c10a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.464+0000 7f7e240c7700 1 --2- 172.21.15.106:0/1272786659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c10a9b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e180098a0 tx=0x7f7e18009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:37.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.464+0000 7f7e240c7700 1 -- 172.21.15.106:0/1272786659 shutdown_connections 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.464+0000 7f7e240c7700 1 --2- 172.21.15.106:0/1272786659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c10a9b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.464+0000 7f7e240c7700 1 -- 172.21.15.106:0/1272786659 >> 172.21.15.106:0/1272786659 conn(0x7f7e1c101ff0 msgr2=0x7f7e1c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.464+0000 7f7e240c7700 1 -- 172.21.15.106:0/1272786659 shutdown_connections 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.464+0000 7f7e240c7700 1 -- 172.21.15.106:0/1272786659 wait complete. 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.464+0000 7f7e240c7700 1 Processor -- start 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.464+0000 7f7e240c7700 1 -- start start 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e240c7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e240c7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c10bf50 con 0x7f7e1c1085c0 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e21e63700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e21e63700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c19baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54302/0 (socket says 172.21.15.106:54302) 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e21e63700 1 -- 172.21.15.106:0/6832585 learned_addr learned my addr 172.21.15.106:0/6832585 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e21e63700 1 -- 172.21.15.106:0/6832585 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c19bfe0 con 0x7f7e1c1085c0 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e21e63700 1 --2- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c19baa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e180116a0 tx=0x7f7e18011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:37.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e12ffd700 1 -- 172.21.15.106:0/6832585 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e18003a40 con 0x7f7e1c1085c0 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e12ffd700 1 -- 172.21.15.106:0/6832585 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e18003be0 con 0x7f7e1c1085c0 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c19c270 con 0x7f7e1c1085c0 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.465+0000 7f7e12ffd700 1 -- 172.21.15.106:0/6832585 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e1801a750 con 0x7f7e1c1085c0 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.466+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c19c6d0 con 0x7f7e1c1085c0 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.466+0000 7f7e12ffd700 1 -- 172.21.15.106:0/6832585 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f7e18024aa0 con 0x7f7e1c1085c0 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.466+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c069260 con 0x7f7e1c1085c0 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.467+0000 7f7e12ffd700 1 --2- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7e08038f80 0x7f7e0803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.467+0000 7f7e12ffd700 1 -- 172.21.15.106:0/6832585 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7e1804d470 con 0x7f7e1c1085c0 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.467+0000 7f7e21662700 1 -- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7e08038f80 msgr2=0x7f7e0803b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/305853075 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.467+0000 7f7e21662700 1 --2- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7e08038f80 0x7f7e0803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.471+0000 7f7e12ffd700 1 -- 172.21.15.106:0/6832585 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7e18011e20 con 0x7f7e1c1085c0 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.668+0000 7f7e21662700 1 -- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7e08038f80 msgr2=0x7f7e0803b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/305853075 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.668+0000 7f7e21662700 1 --2- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7e08038f80 0x7f7e0803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T02:01:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.731+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7e1c19f080 con 0x7f7e1c1085c0 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.732+0000 7f7e12ffd700 1 -- 172.21.15.106:0/6832585 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f7e1802d770 con 0x7f7e1c1085c0 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.734+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7e08038f80 msgr2=0x7f7e0803b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.734+0000 7f7e240c7700 1 --2- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7e08038f80 0x7f7e0803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.734+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 msgr2=0x7f7e1c19baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.734+0000 7f7e240c7700 1 --2- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c19baa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e180116a0 tx=0x7f7e18011ba0 comp rx=0 tx=0).stop 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.734+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 shutdown_connections 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.734+0000 7f7e240c7700 1 --2- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7f7e08038f80 0x7f7e0803b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.734+0000 7f7e240c7700 1 --2- 172.21.15.106:0/6832585 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e1c1085c0 0x7f7e1c19baa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.734+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 >> 172.21.15.106:0/6832585 conn(0x7f7e1c101ff0 msgr2=0x7f7e1c102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.735+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 shutdown_connections 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:36.735+0000 7f7e240c7700 1 -- 172.21.15.106:0/6832585 wait complete. 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for the mgr to restart... 2024-03-28T02:01:37.485 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr epoch 9... 2024-03-28T02:01:37.699 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:37 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/6832585' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T02:01:37.699 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:37.450+0000 7efc5b7bd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T02:01:37.700 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:37.565+0000 7efc5b7bd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T02:01:37.700 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:37.651+0000 7efc5b7bd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T02:01:38.084 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:37.749+0000 7efc5b7bd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T02:01:38.084 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:37.927+0000 7efc5b7bd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T02:01:39.324 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:38 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:38.875+0000 7efc5b7bd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T02:01:39.325 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:39 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:39.003+0000 7efc5b7bd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T02:01:39.325 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:39 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:39.330+0000 7efc5b7bd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T02:01:39.584 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:39 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:39.418+0000 7efc5b7bd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T02:01:40.078 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:39 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:39.614+0000 7efc5b7bd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:01:40.078 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:39 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:39.799+0000 7efc5b7bd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T02:01:40.332 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:40 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:40.084+0000 7efc5b7bd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:01:40.333 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:40 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:40.338+0000 7efc5b7bd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T02:01:40.584 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:40 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:40.575+0000 7efc5b7bd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T02:01:41.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:41 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:41.376+0000 7efc5b7bd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T02:01:42.248 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:41 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:41.938+0000 7efc5b7bd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T02:01:42.248 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:42.036+0000 7efc5b7bd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T02:01:42.248 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:01:42.253+0000 7efc5b7bd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T02:01:42.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: Active manager daemon x restarted 2024-03-28T02:01:42.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: Activating manager daemon x 2024-03-28T02:01:42.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T02:01:42.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: mgrmap e10: x(active, starting, since 0.00316974s) 2024-03-28T02:01:42.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:01:42.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T02:01:42.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T02:01:42.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T02:01:42.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T02:01:42.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: Manager daemon x is now available 2024-03-28T02:01:42.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:42.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:43.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T02:01:43.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T02:01:43.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:43 smithi106 ceph-mon[31965]: mgrmap e11: x(active, since 1.00572s) 2024-03-28T02:01:44.268 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.926+0000 7fafe58b6700 1 Processor -- start 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.926+0000 7fafe58b6700 1 -- start start 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.926+0000 7fafe58b6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe010aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.926+0000 7fafe58b6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe007bd80 con 0x7fafe0108680 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.926+0000 7fafdeffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe010aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.926+0000 7fafdeffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe010aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:43244/0 (socket says 172.21.15.106:43244) 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.926+0000 7fafdeffd700 1 -- 172.21.15.106:0/3777020704 learned_addr learned my addr 172.21.15.106:0/3777020704 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:44.269 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.926+0000 7fafdeffd700 1 -- 172.21.15.106:0/3777020704 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe010afb0 con 0x7fafe0108680 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.927+0000 7fafdeffd700 1 --2- 172.21.15.106:0/3777020704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe010aa70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fafd00098a0 tx=0x7fafd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0e65d21045384a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.927+0000 7fafddffb700 1 -- 172.21.15.106:0/3777020704 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafd0004160 con 0x7fafe0108680 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.927+0000 7fafddffb700 1 -- 172.21.15.106:0/3777020704 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafd0004300 con 0x7fafe0108680 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.927+0000 7fafddffb700 1 -- 172.21.15.106:0/3777020704 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafd0011b20 con 0x7fafe0108680 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.927+0000 7fafe58b6700 1 -- 172.21.15.106:0/3777020704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 msgr2=0x7fafe010aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.927+0000 7fafe58b6700 1 --2- 172.21.15.106:0/3777020704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe010aa70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fafd00098a0 tx=0x7fafd0009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.928+0000 7fafe58b6700 1 -- 172.21.15.106:0/3777020704 shutdown_connections 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.928+0000 7fafe58b6700 1 --2- 172.21.15.106:0/3777020704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe010aa70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.928+0000 7fafe58b6700 1 -- 172.21.15.106:0/3777020704 >> 172.21.15.106:0/3777020704 conn(0x7fafe0101ff0 msgr2=0x7fafe0104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.928+0000 7fafe58b6700 1 -- 172.21.15.106:0/3777020704 shutdown_connections 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.928+0000 7fafe58b6700 1 -- 172.21.15.106:0/3777020704 wait complete. 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.928+0000 7fafe58b6700 1 Processor -- start 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafe58b6700 1 -- start start 2024-03-28T02:01:44.270 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafe58b6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafe58b6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe010bf50 con 0x7fafe0108680 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafdeffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafdeffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe019ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:43246/0 (socket says 172.21.15.106:43246) 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafdeffd700 1 -- 172.21.15.106:0/1394042001 learned_addr learned my addr 172.21.15.106:0/1394042001 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafdeffd700 1 -- 172.21.15.106:0/1394042001 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe019bf90 con 0x7fafe0108680 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafdeffd700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe019ba50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fafd00117e0 tx=0x7fafd0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafd0003bf0 con 0x7fafe0108680 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.929+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafd001a910 con 0x7fafe0108680 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.930+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafd0024970 con 0x7fafe0108680 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.930+0000 7fafe58b6700 1 -- 172.21.15.106:0/1394042001 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe019c220 con 0x7fafe0108680 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.930+0000 7fafe58b6700 1 -- 172.21.15.106:0/1394042001 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe019c680 con 0x7fafe0108680 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.931+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fafd0022020 con 0x7fafe0108680 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.931+0000 7fafe48b4700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 0x7fafc803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:44.271 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.931+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 --> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafc803baf0 con 0x7fafc8038f20 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.931+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fafd004d250 con 0x7fafe0108680 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.931+0000 7fafde7fc700 1 -- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 msgr2=0x7fafc803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/305853075 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:37.931+0000 7fafde7fc700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 0x7fafc803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:38.132+0000 7fafde7fc700 1 -- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 msgr2=0x7fafc803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/305853075 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:38.132+0000 7fafde7fc700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 0x7fafc803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:38.533+0000 7fafde7fc700 1 -- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 msgr2=0x7fafc803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/305853075 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:38.533+0000 7fafde7fc700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 0x7fafc803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:39.334+0000 7fafde7fc700 1 -- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 msgr2=0x7fafc803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/305853075 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:39.334+0000 7fafde7fc700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 0x7fafc803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:40.935+0000 7fafde7fc700 1 -- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 msgr2=0x7fafc803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/305853075 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:40.935+0000 7fafde7fc700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 0x7fafc803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:42.269+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7fafd001a430 con 0x7fafe0108680 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:42.269+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 msgr2=0x7fafc803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T02:01:44.272 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:42.269+0000 7fafe48b4700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 0x7fafc803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.270+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7fafd004efa0 con 0x7fafe0108680 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.270+0000 7fafe48b4700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafc803c9a0 0x7fafc803ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.270+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafc803baf0 con 0x7fafc803c9a0 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.271+0000 7fafde7fc700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafc803c9a0 0x7fafc803ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.271+0000 7fafde7fc700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafc803c9a0 0x7fafc803ed90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fafd4003a10 tx=0x7fafd40092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.272+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fafc803baf0 con 0x7fafc803c9a0 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.284+0000 7fafe58b6700 1 -- 172.21.15.106:0/1394042001 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fafe0069260 con 0x7fafc803c9a0 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe48b4700 1 -- 172.21.15.106:0/1394042001 <== mgr.14152 v2:172.21.15.106:6800/3501119323 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fafe0069260 con 0x7fafc803c9a0 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 -- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafc803c9a0 msgr2=0x7fafc803ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafc803c9a0 0x7fafc803ed90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fafd4003a10 tx=0x7fafd40092b0 comp rx=0 tx=0).stop 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 -- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 msgr2=0x7fafe019ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe019ba50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fafd00117e0 tx=0x7fafd0011c10 comp rx=0 tx=0).stop 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 -- 172.21.15.106:0/1394042001 shutdown_connections 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafc803c9a0 0x7fafc803ed90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:44.273 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe0108680 0x7fafe019ba50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:44.274 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 --2- 172.21.15.106:0/1394042001 >> [v2:172.21.15.106:6800/305853075,v1:172.21.15.106:6801/305853075] conn(0x7fafc8038f20 0x7fafc803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:44.274 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 -- 172.21.15.106:0/1394042001 >> 172.21.15.106:0/1394042001 conn(0x7fafe0101ff0 msgr2=0x7fafe0102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:44.274 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 -- 172.21.15.106:0/1394042001 shutdown_connections 2024-03-28T02:01:44.274 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:43.285+0000 7fafe58b6700 1 -- 172.21.15.106:0/1394042001 wait complete. 2024-03-28T02:01:44.274 INFO:teuthology.orchestra.run.smithi106.stdout:mgr epoch 9 is available 2024-03-28T02:01:44.274 INFO:teuthology.orchestra.run.smithi106.stdout:Generating a dashboard self-signed certificate... 2024-03-28T02:01:44.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:44 smithi106 ceph-mon[31965]: [28/Mar/2024:02:01:43] ENGINE Bus STARTING 2024-03-28T02:01:44.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:44 smithi106 ceph-mon[31965]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T02:01:44.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:44 smithi106 ceph-mon[31965]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T02:01:44.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:44 smithi106 ceph-mon[31965]: [28/Mar/2024:02:01:43] ENGINE Serving on https://172.21.15.106:7150 2024-03-28T02:01:44.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:44 smithi106 ceph-mon[31965]: [28/Mar/2024:02:01:43] ENGINE Serving on http://172.21.15.106:8765 2024-03-28T02:01:44.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:44 smithi106 ceph-mon[31965]: [28/Mar/2024:02:01:43] ENGINE Bus STARTED 2024-03-28T02:01:44.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:45.666 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.750+0000 7fdbf6fa9700 1 Processor -- start 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.750+0000 7fdbf6fa9700 1 -- start start 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.750+0000 7fdbf6fa9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf0106200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.751+0000 7fdbf6fa9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf01067d0 con 0x7fdbf0105de0 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.751+0000 7fdbf4d45700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf0106200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.751+0000 7fdbf4d45700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf0106200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:43318/0 (socket says 172.21.15.106:43318) 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.751+0000 7fdbf4d45700 1 -- 172.21.15.106:0/2486467818 learned_addr learned my addr 172.21.15.106:0/2486467818 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.751+0000 7fdbf4d45700 1 -- 172.21.15.106:0/2486467818 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf0108d70 con 0x7fdbf0105de0 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.751+0000 7fdbf4d45700 1 --2- 172.21.15.106:0/2486467818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf0106200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40098a0 tx=0x7fdbe4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3d9decf1a3c261 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.751+0000 7fdbef7fe700 1 -- 172.21.15.106:0/2486467818 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdbe4004160 con 0x7fdbf0105de0 2024-03-28T02:01:45.667 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.752+0000 7fdbef7fe700 1 -- 172.21.15.106:0/2486467818 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbe4004300 con 0x7fdbf0105de0 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.752+0000 7fdbef7fe700 1 -- 172.21.15.106:0/2486467818 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdbe4011b20 con 0x7fdbf0105de0 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.752+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2486467818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 msgr2=0x7fdbf0106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.752+0000 7fdbf6fa9700 1 --2- 172.21.15.106:0/2486467818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf0106200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40098a0 tx=0x7fdbe4009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.752+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2486467818 shutdown_connections 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.752+0000 7fdbf6fa9700 1 --2- 172.21.15.106:0/2486467818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf0106200 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.752+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2486467818 >> 172.21.15.106:0/2486467818 conn(0x7fdbf0101710 msgr2=0x7fdbf0103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.752+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2486467818 shutdown_connections 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.752+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2486467818 wait complete. 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.753+0000 7fdbf6fa9700 1 Processor -- start 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.753+0000 7fdbf6fa9700 1 -- start start 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.753+0000 7fdbf6fa9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf019baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.753+0000 7fdbf6fa9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf010c120 con 0x7fdbf0105de0 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbf4d45700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf019baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbf4d45700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf019baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:43330/0 (socket says 172.21.15.106:43330) 2024-03-28T02:01:45.668 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbf4d45700 1 -- 172.21.15.106:0/2510401364 learned_addr learned my addr 172.21.15.106:0/2510401364 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbf4d45700 1 -- 172.21.15.106:0/2510401364 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf019bfe0 con 0x7fdbf0105de0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbf4d45700 1 --2- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf019baa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4000c00 tx=0x7fdbe4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbedffb700 1 -- 172.21.15.106:0/2510401364 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdbe4003b60 con 0x7fdbf0105de0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf019c270 con 0x7fdbf0105de0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbedffb700 1 -- 172.21.15.106:0/2510401364 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbe401a910 con 0x7fdbf0105de0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbedffb700 1 -- 172.21.15.106:0/2510401364 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdbe4024930 con 0x7fdbf0105de0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.754+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf019c6d0 con 0x7fdbf0105de0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.755+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf0069260 con 0x7fdbf0105de0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.755+0000 7fdbedffb700 1 -- 172.21.15.106:0/2510401364 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7fdbe4022020 con 0x7fdbf0105de0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.756+0000 7fdbedffb700 1 --2- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdbe0038ec0 0x7fdbe003b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.756+0000 7fdbedffb700 1 -- 172.21.15.106:0/2510401364 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdbe404cf50 con 0x7fdbf0105de0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.756+0000 7fdbeffff700 1 --2- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdbe0038ec0 0x7fdbe003b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.756+0000 7fdbeffff700 1 --2- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdbe0038ec0 0x7fdbe003b380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc006fd0 tx=0x7fdbdc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:45.669 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.761+0000 7fdbedffb700 1 -- 172.21.15.106:0/2510401364 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdbe402d770 con 0x7fdbf0105de0 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.938+0000 7fdbedffb700 1 -- 172.21.15.106:0/2510401364 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fdbe401aac0 con 0x7fdbf0105de0 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:44.971+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbf01028e0 con 0x7fdbe0038ec0 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.113+0000 7fdbedffb700 1 -- 172.21.15.106:0/2510401364 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fdbf01028e0 con 0x7fdbe0038ec0 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdbe0038ec0 msgr2=0x7fdbe003b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 --2- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdbe0038ec0 0x7fdbe003b380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc006fd0 tx=0x7fdbdc006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 msgr2=0x7fdbf019baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 --2- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf019baa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4000c00 tx=0x7fdbe4011c10 comp rx=0 tx=0).stop 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 shutdown_connections 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 --2- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdbe0038ec0 0x7fdbe003b380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 --2- 172.21.15.106:0/2510401364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdbf0105de0 0x7fdbf019baa0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 >> 172.21.15.106:0/2510401364 conn(0x7fdbf0101710 msgr2=0x7fdbf01021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 shutdown_connections 2024-03-28T02:01:45.670 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:45.115+0000 7fdbf6fa9700 1 -- 172.21.15.106:0/2510401364 wait complete. 2024-03-28T02:01:45.671 INFO:teuthology.orchestra.run.smithi106.stdout:Creating initial admin user... 2024-03-28T02:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:45 smithi106 ceph-mon[31965]: mgrmap e12: x(active, since 2s) 2024-03-28T02:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:45 smithi106 ceph-mon[31965]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:47.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:46 smithi106 ceph-mon[31965]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:47.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:47.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$0qewY.QXOfMP1Ic1Mbqd9Og1UJwnGfgr6HnMru8oiLnl1Y2xy172.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711591306, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.105+0000 7f157c455700 1 Processor -- start 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.105+0000 7f157c455700 1 -- start start 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.105+0000 7f157c455700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157410a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.105+0000 7f157c455700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157407be00 con 0x7f1574108600 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.106+0000 7f157a1f1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157410a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.106+0000 7f157a1f1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157410a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:43334/0 (socket says 172.21.15.106:43334) 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.106+0000 7f157a1f1700 1 -- 172.21.15.106:0/981577971 learned_addr learned my addr 172.21.15.106:0/981577971 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.106+0000 7f157a1f1700 1 -- 172.21.15.106:0/981577971 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157410af30 con 0x7f1574108600 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.106+0000 7f157a1f1700 1 --2- 172.21.15.106:0/981577971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157410a9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15640098a0 tx=0x7f1564009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ffaad6dc562cf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.107+0000 7f15791ef700 1 -- 172.21.15.106:0/981577971 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1564004160 con 0x7f1574108600 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.107+0000 7f15791ef700 1 -- 172.21.15.106:0/981577971 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1564004300 con 0x7f1574108600 2024-03-28T02:01:47.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.107+0000 7f15791ef700 1 -- 172.21.15.106:0/981577971 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1564011b20 con 0x7f1574108600 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.107+0000 7f157c455700 1 -- 172.21.15.106:0/981577971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 msgr2=0x7f157410a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.107+0000 7f157c455700 1 --2- 172.21.15.106:0/981577971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157410a9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15640098a0 tx=0x7f1564009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157c455700 1 -- 172.21.15.106:0/981577971 shutdown_connections 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157c455700 1 --2- 172.21.15.106:0/981577971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157410a9f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157c455700 1 -- 172.21.15.106:0/981577971 >> 172.21.15.106:0/981577971 conn(0x7f1574101ff0 msgr2=0x7f1574104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157c455700 1 -- 172.21.15.106:0/981577971 shutdown_connections 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157c455700 1 -- 172.21.15.106:0/981577971 wait complete. 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157c455700 1 Processor -- start 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157c455700 1 -- start start 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157c455700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157c455700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157410bf40 con 0x7f1574108600 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.108+0000 7f157a1f1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.109+0000 7f157a1f1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157419ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:43340/0 (socket says 172.21.15.106:43340) 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.109+0000 7f157a1f1700 1 -- 172.21.15.106:0/1605812530 learned_addr learned my addr 172.21.15.106:0/1605812530 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:47.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.109+0000 7f157a1f1700 1 -- 172.21.15.106:0/1605812530 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157419bf80 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.109+0000 7f157a1f1700 1 --2- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157419ba40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15640117e0 tx=0x7f1564011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.109+0000 7f156b7fe700 1 -- 172.21.15.106:0/1605812530 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1564003ba0 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.109+0000 7f156b7fe700 1 -- 172.21.15.106:0/1605812530 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f156401a910 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.109+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157419c210 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.109+0000 7f156b7fe700 1 -- 172.21.15.106:0/1605812530 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15640248f0 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.109+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157419c5b0 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.110+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1574069260 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.111+0000 7f156b7fe700 1 -- 172.21.15.106:0/1605812530 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f1564022020 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.111+0000 7f156b7fe700 1 --2- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1560039150 0x7f156003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.111+0000 7f156b7fe700 1 -- 172.21.15.106:0/1605812530 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f15640248f0 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.112+0000 7f15799f0700 1 --2- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1560039150 0x7f156003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.112+0000 7f15799f0700 1 --2- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1560039150 0x7f156003b610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1570006fd0 tx=0x7f1570006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.120+0000 7f156b7fe700 1 -- 172.21.15.106:0/1605812530 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f156401ab30 con 0x7f1574108600 2024-03-28T02:01:47.205 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.346+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1574103150 con 0x7f1560039150 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.570+0000 7f156b7fe700 1 -- 172.21.15.106:0/1605812530 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1574103150 con 0x7f1560039150 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1560039150 msgr2=0x7f156003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 --2- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1560039150 0x7f156003b610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1570006fd0 tx=0x7f1570006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 msgr2=0x7f157419ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 --2- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157419ba40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15640117e0 tx=0x7f1564011ba0 comp rx=0 tx=0).stop 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 shutdown_connections 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 --2- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1560039150 0x7f156003b610 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 --2- 172.21.15.106:0/1605812530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1574108600 0x7f157419ba40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 >> 172.21.15.106:0/1605812530 conn(0x7f1574101ff0 msgr2=0x7f1574102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 shutdown_connections 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:46.572+0000 7f157c455700 1 -- 172.21.15.106:0/1605812530 wait complete. 2024-03-28T02:01:47.206 INFO:teuthology.orchestra.run.smithi106.stdout:Fetching dashboard port number... 2024-03-28T02:01:48.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:47 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3580412204' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T02:01:48.720 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T02:01:48.720 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.670+0000 7f18223b7700 1 Processor -- start 2024-03-28T02:01:48.720 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.670+0000 7f18223b7700 1 -- start start 2024-03-28T02:01:48.720 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.670+0000 7f18223b7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c106e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:48.720 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.671+0000 7f18223b7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c107420 con 0x7f181c106a30 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.671+0000 7f181bfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c106e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.671+0000 7f181bfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c106e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37442/0 (socket says 172.21.15.106:37442) 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.671+0000 7f181bfff700 1 -- 172.21.15.106:0/934656041 learned_addr learned my addr 172.21.15.106:0/934656041 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.671+0000 7f181bfff700 1 -- 172.21.15.106:0/934656041 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c107c30 con 0x7f181c106a30 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.671+0000 7f181bfff700 1 --2- 172.21.15.106:0/934656041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c106e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f180c0098a0 tx=0x7f180c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5fbaaee633e151 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.671+0000 7f181b7fe700 1 -- 172.21.15.106:0/934656041 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f180c004160 con 0x7f181c106a30 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.671+0000 7f181b7fe700 1 -- 172.21.15.106:0/934656041 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f180c004300 con 0x7f181c106a30 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.671+0000 7f181b7fe700 1 -- 172.21.15.106:0/934656041 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f180c011b20 con 0x7f181c106a30 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.672+0000 7f18223b7700 1 -- 172.21.15.106:0/934656041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 msgr2=0x7f181c106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.672+0000 7f18223b7700 1 --2- 172.21.15.106:0/934656041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c106e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f180c0098a0 tx=0x7f180c009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.672+0000 7f18223b7700 1 -- 172.21.15.106:0/934656041 shutdown_connections 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.672+0000 7f18223b7700 1 --2- 172.21.15.106:0/934656041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c106e50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.672+0000 7f18223b7700 1 -- 172.21.15.106:0/934656041 >> 172.21.15.106:0/934656041 conn(0x7f181c101ff0 msgr2=0x7f181c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.672+0000 7f18223b7700 1 -- 172.21.15.106:0/934656041 shutdown_connections 2024-03-28T02:01:48.721 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.672+0000 7f18223b7700 1 -- 172.21.15.106:0/934656041 wait complete. 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f18223b7700 1 Processor -- start 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f18223b7700 1 -- start start 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f18223b7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f18223b7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c10afe0 con 0x7f181c106a30 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f181bfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f181bfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c19aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37446/0 (socket says 172.21.15.106:37446) 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f181bfff700 1 -- 172.21.15.106:0/3580412204 learned_addr learned my addr 172.21.15.106:0/3580412204 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f181bfff700 1 -- 172.21.15.106:0/3580412204 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c19afe0 con 0x7f181c106a30 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f181bfff700 1 --2- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c19aaa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f180c0116a0 tx=0x7f180c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f1819ffb700 1 -- 172.21.15.106:0/3580412204 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f180c003a40 con 0x7f181c106a30 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.673+0000 7f1819ffb700 1 -- 172.21.15.106:0/3580412204 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f180c003be0 con 0x7f181c106a30 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.674+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c19b270 con 0x7f181c106a30 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.674+0000 7f1819ffb700 1 -- 172.21.15.106:0/3580412204 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f180c01a720 con 0x7f181c106a30 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.674+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c19bd90 con 0x7f181c106a30 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.674+0000 7f1819ffb700 1 -- 172.21.15.106:0/3580412204 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f180c01aa40 con 0x7f181c106a30 2024-03-28T02:01:48.722 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.674+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c0637c0 con 0x7f181c106a30 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.675+0000 7f1819ffb700 1 --2- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1808038f20 0x7f180803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.675+0000 7f1819ffb700 1 -- 172.21.15.106:0/3580412204 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f180c04d5d0 con 0x7f181c106a30 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.675+0000 7f1813fff700 1 --2- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1808038f20 0x7f180803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.675+0000 7f1813fff700 1 --2- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1808038f20 0x7f180803b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1804006fd0 tx=0x7f1804006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.680+0000 7f1819ffb700 1 -- 172.21.15.106:0/3580412204 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f180c011e20 con 0x7f181c106a30 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.879+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f181c19c1d0 con 0x7f181c106a30 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.880+0000 7f1819ffb700 1 -- 172.21.15.106:0/3580412204 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f180c00bc70 con 0x7f181c106a30 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1808038f20 msgr2=0x7f180803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 --2- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1808038f20 0x7f180803b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1804006fd0 tx=0x7f1804006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 msgr2=0x7f181c19aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 --2- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c19aaa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f180c0116a0 tx=0x7f180c011ba0 comp rx=0 tx=0).stop 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 shutdown_connections 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 --2- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1808038f20 0x7f180803b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 --2- 172.21.15.106:0/3580412204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f181c106a30 0x7f181c19aaa0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:48.723 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 >> 172.21.15.106:0/3580412204 conn(0x7f181c101ff0 msgr2=0x7f181c102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:48.724 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 shutdown_connections 2024-03-28T02:01:48.724 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:47.882+0000 7f18223b7700 1 -- 172.21.15.106:0/3580412204 wait complete. 2024-03-28T02:01:48.728 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T02:01:48.728 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-03-28T02:01:48.736 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T02:01:48.736 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-28T02:01:48.736 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-03-28T02:01:48.736 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T02:01:48.738 INFO:teuthology.orchestra.run.smithi106.stdout:Ceph Dashboard is now available at: 2024-03-28T02:01:48.738 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:48.738 INFO:teuthology.orchestra.run.smithi106.stdout: URL: https://smithi106.front.sepia.ceph.com:8443/ 2024-03-28T02:01:48.738 INFO:teuthology.orchestra.run.smithi106.stdout: User: admin 2024-03-28T02:01:48.738 INFO:teuthology.orchestra.run.smithi106.stdout: Password: heup2we9id 2024-03-28T02:01:48.738 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:48.738 INFO:teuthology.orchestra.run.smithi106.stdout:Saving cluster configuration to /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config directory 2024-03-28T02:01:48.739 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling autotune for osd_memory_target 2024-03-28T02:01:49.927 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.171+0000 7f0768389700 1 Processor -- start 2024-03-28T02:01:49.927 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.171+0000 7f0768389700 1 -- start start 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.171+0000 7f0768389700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.171+0000 7f0768389700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076007c8a0 con 0x7f076007de70 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.171+0000 7f0766125700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.171+0000 7f0766125700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37448/0 (socket says 172.21.15.106:37448) 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.171+0000 7f0766125700 1 -- 172.21.15.106:0/3084207562 learned_addr learned my addr 172.21.15.106:0/3084207562 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.172+0000 7f0766125700 1 -- 172.21.15.106:0/3084207562 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076007ca20 con 0x7f076007de70 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.172+0000 7f0766125700 1 --2- 172.21.15.106:0/3084207562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076007c2d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f075c0098a0 tx=0x7f075c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f60d8f4213c5f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.172+0000 7f0765123700 1 -- 172.21.15.106:0/3084207562 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f075c004160 con 0x7f076007de70 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.172+0000 7f0765123700 1 -- 172.21.15.106:0/3084207562 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f075c004300 con 0x7f076007de70 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.172+0000 7f0765123700 1 -- 172.21.15.106:0/3084207562 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f075c011b20 con 0x7f076007de70 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.172+0000 7f0768389700 1 -- 172.21.15.106:0/3084207562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 msgr2=0x7f076007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.172+0000 7f0768389700 1 --2- 172.21.15.106:0/3084207562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076007c2d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f075c0098a0 tx=0x7f075c009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0768389700 1 -- 172.21.15.106:0/3084207562 shutdown_connections 2024-03-28T02:01:49.928 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0768389700 1 --2- 172.21.15.106:0/3084207562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076007c2d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0768389700 1 -- 172.21.15.106:0/3084207562 >> 172.21.15.106:0/3084207562 conn(0x7f0760103190 msgr2=0x7f07601055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0768389700 1 -- 172.21.15.106:0/3084207562 shutdown_connections 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0768389700 1 -- 172.21.15.106:0/3084207562 wait complete. 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0768389700 1 Processor -- start 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0768389700 1 -- start start 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0768389700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0768389700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076010e150 con 0x7f076007de70 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.173+0000 7f0766125700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.174+0000 7f0766125700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076019de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37452/0 (socket says 172.21.15.106:37452) 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.174+0000 7f0766125700 1 -- 172.21.15.106:0/3895878752 learned_addr learned my addr 172.21.15.106:0/3895878752 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.174+0000 7f0766125700 1 -- 172.21.15.106:0/3895878752 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076019e380 con 0x7f076007de70 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.174+0000 7f0766125700 1 --2- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076019de40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f075c0116a0 tx=0x7f075c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.174+0000 7f07577fe700 1 -- 172.21.15.106:0/3895878752 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f075c003a40 con 0x7f076007de70 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.174+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076019e610 con 0x7f076007de70 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.174+0000 7f07577fe700 1 -- 172.21.15.106:0/3895878752 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f075c003be0 con 0x7f076007de70 2024-03-28T02:01:49.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.174+0000 7f07577fe700 1 -- 172.21.15.106:0/3895878752 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f075c01a720 con 0x7f076007de70 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.174+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076019eff0 con 0x7f076007de70 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.175+0000 7f07577fe700 1 -- 172.21.15.106:0/3895878752 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f075c01aa20 con 0x7f076007de70 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.175+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07600637c0 con 0x7f076007de70 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.175+0000 7f07577fe700 1 --2- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f074c038fc0 0x7f074c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.175+0000 7f07577fe700 1 -- 172.21.15.106:0/3895878752 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f075c04d790 con 0x7f076007de70 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.175+0000 7f0765924700 1 --2- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f074c038fc0 0x7f074c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.175+0000 7f0765924700 1 --2- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f074c038fc0 0x7f074c03b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0750006fd0 tx=0x7f0750006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.180+0000 7f07577fe700 1 -- 172.21.15.106:0/3895878752 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f075c011e20 con 0x7f076007de70 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.375+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f076019f550 con 0x7f076007de70 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.375+0000 7f07577fe700 1 -- 172.21.15.106:0/3895878752 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f075c00bc40 con 0x7f076007de70 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.377+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f074c038fc0 msgr2=0x7f074c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.377+0000 7f0768389700 1 --2- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f074c038fc0 0x7f074c03b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0750006fd0 tx=0x7f0750006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.377+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 msgr2=0x7f076019de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:49.930 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.377+0000 7f0768389700 1 --2- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076019de40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f075c0116a0 tx=0x7f075c011ba0 comp rx=0 tx=0).stop 2024-03-28T02:01:49.931 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.377+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 shutdown_connections 2024-03-28T02:01:49.931 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.377+0000 7f0768389700 1 --2- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f074c038fc0 0x7f074c03b480 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:49.931 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.377+0000 7f0768389700 1 --2- 172.21.15.106:0/3895878752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f076007de70 0x7f076019de40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:49.931 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.377+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 >> 172.21.15.106:0/3895878752 conn(0x7f0760103190 msgr2=0x7f0760103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:49.931 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.378+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 shutdown_connections 2024-03-28T02:01:49.931 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:49.378+0000 7f0768389700 1 -- 172.21.15.106:0/3895878752 wait complete. 2024-03-28T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:49 smithi106 ceph-mon[31965]: mgrmap e13: x(active, since 6s) 2024-03-28T02:01:51.200 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.330+0000 7fab737a2700 1 Processor -- start 2024-03-28T02:01:51.200 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.330+0000 7fab737a2700 1 -- start start 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.330+0000 7fab737a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c106830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.331+0000 7fab737a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c07bde0 con 0x7fab6c106410 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.331+0000 7fab7153e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.331+0000 7fab7153e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37456/0 (socket says 172.21.15.106:37456) 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.331+0000 7fab7153e700 1 -- 172.21.15.106:0/3318942194 learned_addr learned my addr 172.21.15.106:0/3318942194 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.331+0000 7fab7153e700 1 -- 172.21.15.106:0/3318942194 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c106d70 con 0x7fab6c106410 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.331+0000 7fab7153e700 1 --2- 172.21.15.106:0/3318942194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c106830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fab680098a0 tx=0x7fab68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665ac6cc09f98c34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.331+0000 7fab63fff700 1 -- 172.21.15.106:0/3318942194 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab68004160 con 0x7fab6c106410 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.331+0000 7fab63fff700 1 -- 172.21.15.106:0/3318942194 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab68004300 con 0x7fab6c106410 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.331+0000 7fab63fff700 1 -- 172.21.15.106:0/3318942194 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab68011b20 con 0x7fab6c106410 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.332+0000 7fab737a2700 1 -- 172.21.15.106:0/3318942194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 msgr2=0x7fab6c106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.332+0000 7fab737a2700 1 --2- 172.21.15.106:0/3318942194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c106830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fab680098a0 tx=0x7fab68009bb0 comp rx=0 tx=0).stop 2024-03-28T02:01:51.201 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.332+0000 7fab737a2700 1 -- 172.21.15.106:0/3318942194 shutdown_connections 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.332+0000 7fab737a2700 1 --2- 172.21.15.106:0/3318942194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c106830 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.332+0000 7fab737a2700 1 -- 172.21.15.106:0/3318942194 >> 172.21.15.106:0/3318942194 conn(0x7fab6c101ff0 msgr2=0x7fab6c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.332+0000 7fab737a2700 1 -- 172.21.15.106:0/3318942194 shutdown_connections 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.332+0000 7fab737a2700 1 -- 172.21.15.106:0/3318942194 wait complete. 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.332+0000 7fab737a2700 1 Processor -- start 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab737a2700 1 -- start start 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab737a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab737a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c107e40 con 0x7fab6c106410 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab7153e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab7153e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c19dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37460/0 (socket says 172.21.15.106:37460) 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab7153e700 1 -- 172.21.15.106:0/3387106801 learned_addr learned my addr 172.21.15.106:0/3387106801 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab7153e700 1 -- 172.21.15.106:0/3387106801 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c19e320 con 0x7fab6c106410 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab7153e700 1 --2- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c19dde0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fab680117e0 tx=0x7fab68011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:51.202 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab627fc700 1 -- 172.21.15.106:0/3387106801 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab68003bf0 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab627fc700 1 -- 172.21.15.106:0/3387106801 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab6801a910 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab627fc700 1 -- 172.21.15.106:0/3387106801 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab68024970 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6c19e5b0 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.333+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6c19ef90 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.334+0000 7fab627fc700 1 -- 172.21.15.106:0/3387106801 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fab68022020 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.334+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6c0637c0 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.334+0000 7fab627fc700 1 --2- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab58038fc0 0x7fab5803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.335+0000 7fab627fc700 1 -- 172.21.15.106:0/3387106801 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fab6804d460 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.335+0000 7fab70d3d700 1 --2- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab58038fc0 0x7fab5803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.335+0000 7fab70d3d700 1 --2- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab58038fc0 0x7fab5803b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fab5c006fd0 tx=0x7fab5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.339+0000 7fab627fc700 1 -- 172.21.15.106:0/3387106801 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab68011e20 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.646+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fab6c19fc50 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.648+0000 7fab627fc700 1 -- 172.21.15.106:0/3387106801 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fab6800b3e0 con 0x7fab6c106410 2024-03-28T02:01:51.203 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab58038fc0 msgr2=0x7fab5803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 --2- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab58038fc0 0x7fab5803b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fab5c006fd0 tx=0x7fab5c006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 msgr2=0x7fab6c19dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 --2- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c19dde0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fab680117e0 tx=0x7fab68011c10 comp rx=0 tx=0).stop 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 shutdown_connections 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 --2- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab58038fc0 0x7fab5803b480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 --2- 172.21.15.106:0/3387106801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c106410 0x7fab6c19dde0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 >> 172.21.15.106:0/3387106801 conn(0x7fab6c101ff0 msgr2=0x7fab6c1948d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 shutdown_connections 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr 2024-03-28T02:01:50.651+0000 7fab737a2700 1 -- 172.21.15.106:0/3387106801 wait complete. 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T02:01:51.204 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout: ceph telemetry on 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout:For more information see: 2024-03-28T02:01:51.205 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:51.206 INFO:teuthology.orchestra.run.smithi106.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T02:01:51.206 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:01:51.206 INFO:teuthology.orchestra.run.smithi106.stdout:Bootstrap complete. 2024-03-28T02:01:51.241 INFO:tasks.cephadm:Fetching config... 2024-03-28T02:01:51.241 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:01:51.242 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T02:01:51.257 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T02:01:51.257 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:01:51.257 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T02:01:51.312 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T02:01:51.312 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:01:51.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T02:01:51.380 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T02:01:51.380 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:01:51.380 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T02:01:51.435 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T02:01:51.435 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdANol1AwH56UNCZECPEU81h8ybII4H7kKAwqzKyX8QkOXMukb0F2sh7judVQh5hOLMRyf3MY3pNIyf9EvzjWlEBFP3pKLasdbkbpWP13fcpDVfpsXrF+qBm4BbCn33fYbyulLKnfUFswrKNZxknY8HgTIcK5ZLl6FzZZzaG6iXLQy7GXZkBNI0wqYE/NvEdF84LE57RrycKjByzvUlh+E36BX8ag4h90/4Fsu9eLme6ZSmVYIS7zHdXxd5uBtBU+H39UPnqhAnuTAHmbhGGcDPR+Zr+p8rzDx52wzI0vxh/goT7Uv1vbDu2I6LmpyuM13V2z6KYjU0wId9BsIUEOKVDGt7AzFHM4CMmXIEyI3cFeTWuTM9IAii5Lb0gobPClR/HW64N1psp6xfwTu7zA7YAu6Jovjv4tMoE9Mnv3CkvwbMU70iyT3VpnekLlAjuwZNw4E21G4ic4P5LhhYwnZ/PIbTAL72VxckjlESPm0LPr0D9wctaDh+B0nXat2YOM= ceph-d222a03c-eca6-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T02:01:51.515 INFO:teuthology.orchestra.run.smithi106.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdANol1AwH56UNCZECPEU81h8ybII4H7kKAwqzKyX8QkOXMukb0F2sh7judVQh5hOLMRyf3MY3pNIyf9EvzjWlEBFP3pKLasdbkbpWP13fcpDVfpsXrF+qBm4BbCn33fYbyulLKnfUFswrKNZxknY8HgTIcK5ZLl6FzZZzaG6iXLQy7GXZkBNI0wqYE/NvEdF84LE57RrycKjByzvUlh+E36BX8ag4h90/4Fsu9eLme6ZSmVYIS7zHdXxd5uBtBU+H39UPnqhAnuTAHmbhGGcDPR+Zr+p8rzDx52wzI0vxh/goT7Uv1vbDu2I6LmpyuM13V2z6KYjU0wId9BsIUEOKVDGt7AzFHM4CMmXIEyI3cFeTWuTM9IAii5Lb0gobPClR/HW64N1psp6xfwTu7zA7YAu6Jovjv4tMoE9Mnv3CkvwbMU70iyT3VpnekLlAjuwZNw4E21G4ic4P5LhhYwnZ/PIbTAL72VxckjlESPm0LPr0D9wctaDh+B0nXat2YOM= ceph-d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:01:51.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdANol1AwH56UNCZECPEU81h8ybII4H7kKAwqzKyX8QkOXMukb0F2sh7judVQh5hOLMRyf3MY3pNIyf9EvzjWlEBFP3pKLasdbkbpWP13fcpDVfpsXrF+qBm4BbCn33fYbyulLKnfUFswrKNZxknY8HgTIcK5ZLl6FzZZzaG6iXLQy7GXZkBNI0wqYE/NvEdF84LE57RrycKjByzvUlh+E36BX8ag4h90/4Fsu9eLme6ZSmVYIS7zHdXxd5uBtBU+H39UPnqhAnuTAHmbhGGcDPR+Zr+p8rzDx52wzI0vxh/goT7Uv1vbDu2I6LmpyuM13V2z6KYjU0wId9BsIUEOKVDGt7AzFHM4CMmXIEyI3cFeTWuTM9IAii5Lb0gobPClR/HW64N1psp6xfwTu7zA7YAu6Jovjv4tMoE9Mnv3CkvwbMU70iyT3VpnekLlAjuwZNw4E21G4ic4P5LhhYwnZ/PIbTAL72VxckjlESPm0LPr0D9wctaDh+B0nXat2YOM= ceph-d222a03c-eca6-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T02:01:51.578 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdANol1AwH56UNCZECPEU81h8ybII4H7kKAwqzKyX8QkOXMukb0F2sh7judVQh5hOLMRyf3MY3pNIyf9EvzjWlEBFP3pKLasdbkbpWP13fcpDVfpsXrF+qBm4BbCn33fYbyulLKnfUFswrKNZxknY8HgTIcK5ZLl6FzZZzaG6iXLQy7GXZkBNI0wqYE/NvEdF84LE57RrycKjByzvUlh+E36BX8ag4h90/4Fsu9eLme6ZSmVYIS7zHdXxd5uBtBU+H39UPnqhAnuTAHmbhGGcDPR+Zr+p8rzDx52wzI0vxh/goT7Uv1vbDu2I6LmpyuM13V2z6KYjU0wId9BsIUEOKVDGt7AzFHM4CMmXIEyI3cFeTWuTM9IAii5Lb0gobPClR/HW64N1psp6xfwTu7zA7YAu6Jovjv4tMoE9Mnv3CkvwbMU70iyT3VpnekLlAjuwZNw4E21G4ic4P5LhhYwnZ/PIbTAL72VxckjlESPm0LPr0D9wctaDh+B0nXat2YOM= ceph-d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:01:51.593 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdANol1AwH56UNCZECPEU81h8ybII4H7kKAwqzKyX8QkOXMukb0F2sh7judVQh5hOLMRyf3MY3pNIyf9EvzjWlEBFP3pKLasdbkbpWP13fcpDVfpsXrF+qBm4BbCn33fYbyulLKnfUFswrKNZxknY8HgTIcK5ZLl6FzZZzaG6iXLQy7GXZkBNI0wqYE/NvEdF84LE57RrycKjByzvUlh+E36BX8ag4h90/4Fsu9eLme6ZSmVYIS7zHdXxd5uBtBU+H39UPnqhAnuTAHmbhGGcDPR+Zr+p8rzDx52wzI0vxh/goT7Uv1vbDu2I6LmpyuM13V2z6KYjU0wId9BsIUEOKVDGt7AzFHM4CMmXIEyI3cFeTWuTM9IAii5Lb0gobPClR/HW64N1psp6xfwTu7zA7YAu6Jovjv4tMoE9Mnv3CkvwbMU70iyT3VpnekLlAjuwZNw4E21G4ic4P5LhhYwnZ/PIbTAL72VxckjlESPm0LPr0D9wctaDh+B0nXat2YOM= ceph-d222a03c-eca6-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T02:01:51.642 INFO:teuthology.orchestra.run.smithi178.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdANol1AwH56UNCZECPEU81h8ybII4H7kKAwqzKyX8QkOXMukb0F2sh7judVQh5hOLMRyf3MY3pNIyf9EvzjWlEBFP3pKLasdbkbpWP13fcpDVfpsXrF+qBm4BbCn33fYbyulLKnfUFswrKNZxknY8HgTIcK5ZLl6FzZZzaG6iXLQy7GXZkBNI0wqYE/NvEdF84LE57RrycKjByzvUlh+E36BX8ag4h90/4Fsu9eLme6ZSmVYIS7zHdXxd5uBtBU+H39UPnqhAnuTAHmbhGGcDPR+Zr+p8rzDx52wzI0vxh/goT7Uv1vbDu2I6LmpyuM13V2z6KYjU0wId9BsIUEOKVDGt7AzFHM4CMmXIEyI3cFeTWuTM9IAii5Lb0gobPClR/HW64N1psp6xfwTu7zA7YAu6Jovjv4tMoE9Mnv3CkvwbMU70iyT3VpnekLlAjuwZNw4E21G4ic4P5LhhYwnZ/PIbTAL72VxckjlESPm0LPr0D9wctaDh+B0nXat2YOM= ceph-d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:01:51.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T02:01:51.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:51 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3387106801' entity='client.admin' 2024-03-28T02:01:51.892 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.518+0000 7f7eff865700 1 -- 172.21.15.106:0/3714183783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 msgr2=0x7f7ef8101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.518+0000 7f7eff865700 1 --2- 172.21.15.106:0/3714183783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 0x7f7ef8101ff0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009910 tx=0x7f7ef4009c20 comp rx=0 tx=0).stop 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.518+0000 7f7eff865700 1 -- 172.21.15.106:0/3714183783 shutdown_connections 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.518+0000 7f7eff865700 1 --2- 172.21.15.106:0/3714183783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 0x7f7ef8101ff0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.518+0000 7f7eff865700 1 -- 172.21.15.106:0/3714183783 >> 172.21.15.106:0/3714183783 conn(0x7f7ef80fd170 msgr2=0x7f7ef80ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.518+0000 7f7eff865700 1 -- 172.21.15.106:0/3714183783 shutdown_connections 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.518+0000 7f7eff865700 1 -- 172.21.15.106:0/3714183783 wait complete. 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.519+0000 7f7eff865700 1 Processor -- start 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.519+0000 7f7eff865700 1 -- start start 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.519+0000 7f7eff865700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 0x7f7ef8199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.519+0000 7f7eff865700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8107c60 con 0x7f7ef8101bd0 2024-03-28T02:01:54.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.519+0000 7f7efd601700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 0x7f7ef8199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:54.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.519+0000 7f7efd601700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 0x7f7ef8199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37468/0 (socket says 172.21.15.106:37468) 2024-03-28T02:01:54.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.519+0000 7f7efd601700 1 -- 172.21.15.106:0/1623594537 learned_addr learned my addr 172.21.15.106:0/1623594537 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:54.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.519+0000 7f7efd601700 1 -- 172.21.15.106:0/1623594537 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef8199a90 con 0x7f7ef8101bd0 2024-03-28T02:01:54.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.519+0000 7f7efd601700 1 --2- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 0x7f7ef8199550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009fd0 tx=0x7f7ef4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:54.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.520+0000 7f7eee7fc700 1 -- 172.21.15.106:0/1623594537 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ef4027070 con 0x7f7ef8101bd0 2024-03-28T02:01:54.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.520+0000 7f7eee7fc700 1 -- 172.21.15.106:0/1623594537 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ef400be40 con 0x7f7ef8101bd0 2024-03-28T02:01:54.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.520+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef8199d20 con 0x7f7ef8101bd0 2024-03-28T02:01:54.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.520+0000 7f7eee7fc700 1 -- 172.21.15.106:0/1623594537 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ef400f460 con 0x7f7ef8101bd0 2024-03-28T02:01:54.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.520+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef819ca10 con 0x7f7ef8101bd0 2024-03-28T02:01:54.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.520+0000 7f7eee7fc700 1 -- 172.21.15.106:0/1623594537 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f7ef402a070 con 0x7f7ef8101bd0 2024-03-28T02:01:54.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.521+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef81083e0 con 0x7f7ef8101bd0 2024-03-28T02:01:54.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.521+0000 7f7eee7fc700 1 --2- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7ee4039020 0x7f7ee403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:54.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.521+0000 7f7eee7fc700 1 -- 172.21.15.106:0/1623594537 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ef401e030 con 0x7f7ef8101bd0 2024-03-28T02:01:54.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.521+0000 7f7efce00700 1 --2- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7ee4039020 0x7f7ee403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:54.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.521+0000 7f7efce00700 1 --2- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7ee4039020 0x7f7ee403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8006fd0 tx=0x7f7ee8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:54.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.526+0000 7f7eee7fc700 1 -- 172.21.15.106:0/1623594537 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7ef4013070 con 0x7f7ef8101bd0 2024-03-28T02:01:54.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.729+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f7ef802db40 con 0x7f7ef8101bd0 2024-03-28T02:01:54.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.732+0000 7f7eee7fc700 1 -- 172.21.15.106:0/1623594537 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7ef402a360 con 0x7f7ef8101bd0 2024-03-28T02:01:54.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.734+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7ee4039020 msgr2=0x7f7ee403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:54.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.734+0000 7f7eff865700 1 --2- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7ee4039020 0x7f7ee403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8006fd0 tx=0x7f7ee8006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:54.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.735+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 msgr2=0x7f7ef8199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:54.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.735+0000 7f7eff865700 1 --2- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 0x7f7ef8199550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009fd0 tx=0x7f7ef4005050 comp rx=0 tx=0).stop 2024-03-28T02:01:54.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.735+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 shutdown_connections 2024-03-28T02:01:54.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.735+0000 7f7eff865700 1 --2- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7ee4039020 0x7f7ee403b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:54.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.735+0000 7f7eff865700 1 --2- 172.21.15.106:0/1623594537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef8101bd0 0x7f7ef8199550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:54.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.735+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 >> 172.21.15.106:0/1623594537 conn(0x7f7ef80fd170 msgr2=0x7f7ef80ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:54.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.735+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 shutdown_connections 2024-03-28T02:01:54.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:54.735+0000 7f7eff865700 1 -- 172.21.15.106:0/1623594537 wait complete. 2024-03-28T02:01:54.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:54 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:54.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:54 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:54.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:54 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:01:54.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:54 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:54.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:54 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:54.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:54 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:55.363 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T02:01:55.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T02:01:55.971 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:55 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1623594537' entity='client.admin' 2024-03-28T02:01:55.971 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:55.972 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:56.015 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:01:57.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.535+0000 7f804891a700 1 -- 172.21.15.106:0/1880198836 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 msgr2=0x7f8044101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:57.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.535+0000 7f804891a700 1 --2- 172.21.15.106:0/1880198836 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 0x7f8044101fe0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8034009910 tx=0x7f8034009c20 comp rx=0 tx=0).stop 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.535+0000 7f804891a700 1 -- 172.21.15.106:0/1880198836 shutdown_connections 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.535+0000 7f804891a700 1 --2- 172.21.15.106:0/1880198836 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 0x7f8044101fe0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.535+0000 7f804891a700 1 -- 172.21.15.106:0/1880198836 >> 172.21.15.106:0/1880198836 conn(0x7f80440fd180 msgr2=0x7f80440ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.535+0000 7f804891a700 1 -- 172.21.15.106:0/1880198836 shutdown_connections 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.535+0000 7f804891a700 1 -- 172.21.15.106:0/1880198836 wait complete. 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.536+0000 7f804891a700 1 Processor -- start 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.536+0000 7f804891a700 1 -- start start 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.536+0000 7f804891a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 0x7f8044197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.536+0000 7f804891a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8044107c50 con 0x7f8044101bc0 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.536+0000 7f804259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 0x7f8044197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.536+0000 7f804259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 0x7f8044197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50812/0 (socket says 172.21.15.106:50812) 2024-03-28T02:01:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.536+0000 7f804259c700 1 -- 172.21.15.106:0/2146818952 learned_addr learned my addr 172.21.15.106:0/2146818952 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:01:57.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.537+0000 7f804259c700 1 -- 172.21.15.106:0/2146818952 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8044197840 con 0x7f8044101bc0 2024-03-28T02:01:57.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.537+0000 7f804259c700 1 --2- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 0x7f8044197300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8034006010 tx=0x7f8034005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:57.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.537+0000 7f80337fe700 1 -- 172.21.15.106:0/2146818952 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8034027070 con 0x7f8044101bc0 2024-03-28T02:01:57.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.537+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8044197ad0 con 0x7f8044101bc0 2024-03-28T02:01:57.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.537+0000 7f80337fe700 1 -- 172.21.15.106:0/2146818952 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f803400be40 con 0x7f8044101bc0 2024-03-28T02:01:57.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.537+0000 7f80337fe700 1 -- 172.21.15.106:0/2146818952 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f803400f460 con 0x7f8044101bc0 2024-03-28T02:01:57.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.537+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f804419a7c0 con 0x7f8044101bc0 2024-03-28T02:01:57.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.538+0000 7f80337fe700 1 -- 172.21.15.106:0/2146818952 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f803402a070 con 0x7f8044101bc0 2024-03-28T02:01:57.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.538+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80441083d0 con 0x7f8044101bc0 2024-03-28T02:01:57.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.538+0000 7f80337fe700 1 --2- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f802c038fc0 0x7f802c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:01:57.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.538+0000 7f80337fe700 1 -- 172.21.15.106:0/2146818952 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f803401e030 con 0x7f8044101bc0 2024-03-28T02:01:57.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.538+0000 7f8041d9b700 1 --2- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f802c038fc0 0x7f802c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:01:57.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.538+0000 7f8041d9b700 1 --2- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f802c038fc0 0x7f802c03b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8038006fd0 tx=0x7f8038006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:01:57.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.543+0000 7f80337fe700 1 -- 172.21.15.106:0/2146818952 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8034013070 con 0x7f8044101bc0 2024-03-28T02:01:57.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.746+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f804402db40 con 0x7f802c038fc0 2024-03-28T02:01:57.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.749+0000 7f80337fe700 1 -- 172.21.15.106:0/2146818952 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f804402db40 con 0x7f802c038fc0 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f802c038fc0 msgr2=0x7f802c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 --2- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f802c038fc0 0x7f802c03b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8038006fd0 tx=0x7f8038006e40 comp rx=0 tx=0).stop 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 msgr2=0x7f8044197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 --2- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 0x7f8044197300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8034006010 tx=0x7f8034005050 comp rx=0 tx=0).stop 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 shutdown_connections 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 --2- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f802c038fc0 0x7f802c03b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 --2- 172.21.15.106:0/2146818952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8044101bc0 0x7f8044197300 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 >> 172.21.15.106:0/2146818952 conn(0x7f80440fd180 msgr2=0x7f80440fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 shutdown_connections 2024-03-28T02:01:57.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:01:57.751+0000 7f804891a700 1 -- 172.21.15.106:0/2146818952 wait complete. 2024-03-28T02:01:58.392 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-03-28T02:01:58.392 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T02:01:58.392 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-03-28T02:01:58.411 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T02:01:58.411 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:01:58.468 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-03-28T02:01:58.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch host add smithi153 2024-03-28T02:01:58.708 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:01:58.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:58.744 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:01:58.744 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:01:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:00.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.221+0000 7f5107534700 1 -- 172.21.15.106:0/1128851615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 msgr2=0x7f5100106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:00.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.221+0000 7f5107534700 1 --2- 172.21.15.106:0/1128851615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 0x7f5100106e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009910 tx=0x7f50f0009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:00.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.221+0000 7f5107534700 1 -- 172.21.15.106:0/1128851615 shutdown_connections 2024-03-28T02:02:00.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.221+0000 7f5107534700 1 --2- 172.21.15.106:0/1128851615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 0x7f5100106e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:00.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.221+0000 7f5107534700 1 -- 172.21.15.106:0/1128851615 >> 172.21.15.106:0/1128851615 conn(0x7f51000fc540 msgr2=0x7f51000fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:00.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.221+0000 7f5107534700 1 -- 172.21.15.106:0/1128851615 shutdown_connections 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.221+0000 7f5107534700 1 -- 172.21.15.106:0/1128851615 wait complete. 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.222+0000 7f5107534700 1 Processor -- start 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.222+0000 7f5107534700 1 -- start start 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.222+0000 7f5107534700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 0x7f5100198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.222+0000 7f5107534700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510010b090 con 0x7f5100106a50 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.222+0000 7f51052d0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 0x7f5100198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.222+0000 7f51052d0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 0x7f5100198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50834/0 (socket says 172.21.15.106:50834) 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.222+0000 7f51052d0700 1 -- 172.21.15.106:0/1565143367 learned_addr learned my addr 172.21.15.106:0/1565143367 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.222+0000 7f51052d0700 1 -- 172.21.15.106:0/1565143367 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5100198a70 con 0x7f5100106a50 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.222+0000 7f51052d0700 1 --2- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 0x7f5100198530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f50f0006010 tx=0x7f50f0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.223+0000 7f50f67fc700 1 -- 172.21.15.106:0/1565143367 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50f0026070 con 0x7f5100106a50 2024-03-28T02:02:00.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.223+0000 7f50f67fc700 1 -- 172.21.15.106:0/1565143367 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50f000bbd0 con 0x7f5100106a50 2024-03-28T02:02:00.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.223+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5100198d60 con 0x7f5100106a50 2024-03-28T02:02:00.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.223+0000 7f50f67fc700 1 -- 172.21.15.106:0/1565143367 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50f0003910 con 0x7f5100106a50 2024-03-28T02:02:00.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.223+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5100199220 con 0x7f5100106a50 2024-03-28T02:02:00.220 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.223+0000 7f50f67fc700 1 -- 172.21.15.106:0/1565143367 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f50f0029070 con 0x7f5100106a50 2024-03-28T02:02:00.220 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.224+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5100102190 con 0x7f5100106a50 2024-03-28T02:02:00.220 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.224+0000 7f50f67fc700 1 --2- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f50ec038fc0 0x7f50ec03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:00.220 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.224+0000 7f50f67fc700 1 -- 172.21.15.106:0/1565143367 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f50f001e070 con 0x7f5100106a50 2024-03-28T02:02:00.220 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.224+0000 7f5104acf700 1 --2- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f50ec038fc0 0x7f50ec03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:00.220 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.224+0000 7f5104acf700 1 --2- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f50ec038fc0 0x7f50ec03b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50fc006fd0 tx=0x7f50fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:00.223 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.229+0000 7f50f67fc700 1 -- 172.21.15.106:0/1565143367 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f50f0013070 con 0x7f5100106a50 2024-03-28T02:02:00.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:00.429+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) v1 -- 0x7f510006a5b0 con 0x7f50ec038fc0 2024-03-28T02:02:00.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:00 smithi106 ceph-mon[31965]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:01.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:01 smithi106 ceph-mon[31965]: Deploying cephadm binary to smithi153 2024-03-28T02:02:03.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:03 smithi106 ceph-mon[31965]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.582+0000 7f50f67fc700 1 -- 172.21.15.106:0/1565143367 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f510006a5b0 con 0x7f50ec038fc0 2024-03-28T02:02:04.577 INFO:teuthology.orchestra.run.smithi106.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-03-28T02:02:04.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.584+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f50ec038fc0 msgr2=0x7f50ec03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:04.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.584+0000 7f5107534700 1 --2- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f50ec038fc0 0x7f50ec03b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50fc006fd0 tx=0x7f50fc006e40 comp rx=0 tx=0).stop 2024-03-28T02:02:04.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.584+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 msgr2=0x7f5100198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:04.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.584+0000 7f5107534700 1 --2- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 0x7f5100198530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f50f0006010 tx=0x7f50f0005030 comp rx=0 tx=0).stop 2024-03-28T02:02:04.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.584+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 shutdown_connections 2024-03-28T02:02:04.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.585+0000 7f5107534700 1 --2- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f50ec038fc0 0x7f50ec03b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:04.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.585+0000 7f5107534700 1 --2- 172.21.15.106:0/1565143367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5100106a50 0x7f5100198530 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:04.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.585+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 >> 172.21.15.106:0/1565143367 conn(0x7f51000fc540 msgr2=0x7f51000fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:04.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.585+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 shutdown_connections 2024-03-28T02:02:04.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:04.585+0000 7f5107534700 1 -- 172.21.15.106:0/1565143367 wait complete. 2024-03-28T02:02:05.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T02:02:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:02:05.485 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:05 smithi106 ceph-mon[31965]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:05.485 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:05 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:05.485 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:05 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:05.485 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:05 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:06.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:06 smithi106 ceph-mon[31965]: Added host smithi153 2024-03-28T02:02:06.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:06.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.710+0000 7fa87cabd700 1 -- 172.21.15.106:0/1095517319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 msgr2=0x7fa878103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:06.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.710+0000 7fa87cabd700 1 --2- 172.21.15.106:0/1095517319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 0x7fa878103110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa868009910 tx=0x7fa868009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:06.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.710+0000 7fa87cabd700 1 -- 172.21.15.106:0/1095517319 shutdown_connections 2024-03-28T02:02:06.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.710+0000 7fa87cabd700 1 --2- 172.21.15.106:0/1095517319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 0x7fa878103110 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:06.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.710+0000 7fa87cabd700 1 -- 172.21.15.106:0/1095517319 >> 172.21.15.106:0/1095517319 conn(0x7fa8780fc560 msgr2=0x7fa8780fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:06.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.710+0000 7fa87cabd700 1 -- 172.21.15.106:0/1095517319 shutdown_connections 2024-03-28T02:02:06.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.710+0000 7fa87cabd700 1 -- 172.21.15.106:0/1095517319 wait complete. 2024-03-28T02:02:06.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.711+0000 7fa87cabd700 1 Processor -- start 2024-03-28T02:02:06.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.711+0000 7fa87cabd700 1 -- start start 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.711+0000 7fa87cabd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 0x7fa87819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.711+0000 7fa87cabd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87810c220 con 0x7fa878102d30 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.711+0000 7fa87659c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 0x7fa87819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.711+0000 7fa87659c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 0x7fa87819d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50856/0 (socket says 172.21.15.106:50856) 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.711+0000 7fa87659c700 1 -- 172.21.15.106:0/1034175414 learned_addr learned my addr 172.21.15.106:0/1034175414 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.711+0000 7fa87659c700 1 -- 172.21.15.106:0/1034175414 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87819d910 con 0x7fa878102d30 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.711+0000 7fa87659c700 1 --2- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 0x7fa87819d3d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa868006010 tx=0x7fa868005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.712+0000 7fa8677fe700 1 -- 172.21.15.106:0/1034175414 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa868026070 con 0x7fa878102d30 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.712+0000 7fa8677fe700 1 -- 172.21.15.106:0/1034175414 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa86800bc30 con 0x7fa878102d30 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.712+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87819dba0 con 0x7fa878102d30 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.712+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa878197590 con 0x7fa878102d30 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.712+0000 7fa8677fe700 1 -- 172.21.15.106:0/1034175414 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa868003ab0 con 0x7fa878102d30 2024-03-28T02:02:06.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.712+0000 7fa8677fe700 1 -- 172.21.15.106:0/1034175414 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fa868029070 con 0x7fa878102d30 2024-03-28T02:02:06.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.713+0000 7fa8677fe700 1 --2- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa860039020 0x7fa86003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:06.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.713+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87806a860 con 0x7fa878102d30 2024-03-28T02:02:06.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.713+0000 7fa8677fe700 1 -- 172.21.15.106:0/1034175414 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa86801e070 con 0x7fa878102d30 2024-03-28T02:02:06.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.713+0000 7fa875d9b700 1 --2- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa860039020 0x7fa86003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:06.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.713+0000 7fa875d9b700 1 --2- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa860039020 0x7fa86003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa86c006fd0 tx=0x7fa86c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:06.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.718+0000 7fa8677fe700 1 -- 172.21.15.106:0/1034175414 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa868013070 con 0x7fa878102d30 2024-03-28T02:02:06.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.920+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa878198ac0 con 0x7fa860039020 2024-03-28T02:02:06.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.922+0000 7fa8677fe700 1 -- 172.21.15.106:0/1034175414 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fa878198ac0 con 0x7fa860039020 2024-03-28T02:02:06.916 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:02:06.916 INFO:teuthology.orchestra.run.smithi106.stdout:[{"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-03-28T02:02:06.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa860039020 msgr2=0x7fa86003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:06.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 --2- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa860039020 0x7fa86003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa86c006fd0 tx=0x7fa86c006e40 comp rx=0 tx=0).stop 2024-03-28T02:02:06.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 msgr2=0x7fa87819d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:06.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 --2- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 0x7fa87819d3d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa868006010 tx=0x7fa868005050 comp rx=0 tx=0).stop 2024-03-28T02:02:06.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 shutdown_connections 2024-03-28T02:02:06.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 --2- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa860039020 0x7fa86003b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:06.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 --2- 172.21.15.106:0/1034175414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa878102d30 0x7fa87819d3d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:06.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 >> 172.21.15.106:0/1034175414 conn(0x7fa8780fc560 msgr2=0x7fa878105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:06.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 shutdown_connections 2024-03-28T02:02:06.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:06.924+0000 7fa87cabd700 1 -- 172.21.15.106:0/1034175414 wait complete. 2024-03-28T02:02:07.539 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:07 smithi106 ceph-mon[31965]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:07.539 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:07 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:07.540 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi178 2024-03-28T02:02:07.541 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T02:02:07.541 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.conf 2024-03-28T02:02:07.559 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T02:02:07.559 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:02:07.617 INFO:tasks.cephadm:Adding host smithi178 to orchestrator... 2024-03-28T02:02:07.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch host add smithi178 2024-03-28T02:02:07.850 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:02:08.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:08 smithi106 ceph-mon[31965]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:02:09.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.342+0000 7f9bb15c9700 1 -- 172.21.15.106:0/1527842331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 msgr2=0x7f9bac106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:09.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.342+0000 7f9bb15c9700 1 --2- 172.21.15.106:0/1527842331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 0x7f9bac106e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009910 tx=0x7f9ba0009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:09.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.342+0000 7f9bb15c9700 1 -- 172.21.15.106:0/1527842331 shutdown_connections 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.342+0000 7f9bb15c9700 1 --2- 172.21.15.106:0/1527842331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 0x7f9bac106e50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.342+0000 7f9bb15c9700 1 -- 172.21.15.106:0/1527842331 >> 172.21.15.106:0/1527842331 conn(0x7f9bac0fc5a0 msgr2=0x7f9bac0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.342+0000 7f9bb15c9700 1 -- 172.21.15.106:0/1527842331 shutdown_connections 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.342+0000 7f9bb15c9700 1 -- 172.21.15.106:0/1527842331 wait complete. 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.342+0000 7f9bb15c9700 1 Processor -- start 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.343+0000 7f9bb15c9700 1 -- start start 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.343+0000 7f9bb15c9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 0x7f9bac1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.343+0000 7f9bb15c9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac10b0b0 con 0x7f9bac106a70 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.343+0000 7f9baaffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 0x7f9bac1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.343+0000 7f9baaffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 0x7f9bac1976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:59212/0 (socket says 172.21.15.106:59212) 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.343+0000 7f9baaffd700 1 -- 172.21.15.106:0/573925662 learned_addr learned my addr 172.21.15.106:0/573925662 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.343+0000 7f9baaffd700 1 -- 172.21.15.106:0/573925662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac197be0 con 0x7f9bac106a70 2024-03-28T02:02:09.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.343+0000 7f9baaffd700 1 --2- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 0x7f9bac1976a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0005850 tx=0x7f9ba0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:09.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.343+0000 7f9b93fff700 1 -- 172.21.15.106:0/573925662 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ba0026070 con 0x7f9bac106a70 2024-03-28T02:02:09.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.344+0000 7f9b93fff700 1 -- 172.21.15.106:0/573925662 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ba000bbd0 con 0x7f9bac106a70 2024-03-28T02:02:09.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.344+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac197ed0 con 0x7f9bac106a70 2024-03-28T02:02:09.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.344+0000 7f9b93fff700 1 -- 172.21.15.106:0/573925662 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ba0003970 con 0x7f9bac106a70 2024-03-28T02:02:09.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.344+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac19bf70 con 0x7f9bac106a70 2024-03-28T02:02:09.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.344+0000 7f9b93fff700 1 -- 172.21.15.106:0/573925662 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f9ba0029070 con 0x7f9bac106a70 2024-03-28T02:02:09.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.345+0000 7f9b93fff700 1 --2- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b94039020 0x7f9b9403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:09.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.345+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac1021b0 con 0x7f9bac106a70 2024-03-28T02:02:09.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.345+0000 7f9b93fff700 1 -- 172.21.15.106:0/573925662 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9ba001e070 con 0x7f9bac106a70 2024-03-28T02:02:09.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.345+0000 7f9baa7fc700 1 --2- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b94039020 0x7f9b9403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:09.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.345+0000 7f9baa7fc700 1 --2- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b94039020 0x7f9b9403b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c006fd0 tx=0x7f9b9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:09.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.350+0000 7f9b93fff700 1 -- 172.21.15.106:0/573925662 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ba0013070 con 0x7f9bac106a70 2024-03-28T02:02:09.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:09.547+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bac198a00 con 0x7f9b94039020 2024-03-28T02:02:09.542 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:09 smithi106 ceph-mon[31965]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: Deploying cephadm binary to smithi178 2024-03-28T02:02:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:11.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:11 smithi106 ceph-mon[31965]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:12.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:12 smithi106 ceph-mon[31965]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:02:13.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:13 smithi106 ceph-mon[31965]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:13.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.760+0000 7f9b93fff700 1 -- 172.21.15.106:0/573925662 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9bac198a00 con 0x7f9b94039020 2024-03-28T02:02:13.755 INFO:teuthology.orchestra.run.smithi106.stdout:Added host 'smithi178' with addr '172.21.15.178' 2024-03-28T02:02:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.762+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b94039020 msgr2=0x7f9b9403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.762+0000 7f9bb15c9700 1 --2- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b94039020 0x7f9b9403b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c006fd0 tx=0x7f9b9c006e40 comp rx=0 tx=0).stop 2024-03-28T02:02:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.762+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 msgr2=0x7f9bac1976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.762+0000 7f9bb15c9700 1 --2- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 0x7f9bac1976a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0005850 tx=0x7f9ba0005030 comp rx=0 tx=0).stop 2024-03-28T02:02:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.762+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 shutdown_connections 2024-03-28T02:02:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.762+0000 7f9bb15c9700 1 --2- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b94039020 0x7f9b9403b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.762+0000 7f9bb15c9700 1 --2- 172.21.15.106:0/573925662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac106a70 0x7f9bac1976a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.762+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 >> 172.21.15.106:0/573925662 conn(0x7f9bac0fc5a0 msgr2=0x7f9bac104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.763+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 shutdown_connections 2024-03-28T02:02:13.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:13.763+0000 7f9bb15c9700 1 -- 172.21.15.106:0/573925662 wait complete. 2024-03-28T02:02:14.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T02:02:14.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:14 smithi106 ceph-mon[31965]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:02:14.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:14.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:14.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:14.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:14.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:14.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:14.655 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:02:15.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:15 smithi106 ceph-mon[31965]: Added host smithi178 2024-03-28T02:02:15.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:15 smithi106 ceph-mon[31965]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:15.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:15 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:16.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.125+0000 7fb1a19d9700 1 -- 172.21.15.106:0/2202750063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 msgr2=0x7fb19c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:16.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.125+0000 7fb1a19d9700 1 --2- 172.21.15.106:0/2202750063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 0x7fb19c1031b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009910 tx=0x7fb18c009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:16.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.125+0000 7fb1a19d9700 1 -- 172.21.15.106:0/2202750063 shutdown_connections 2024-03-28T02:02:16.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.125+0000 7fb1a19d9700 1 --2- 172.21.15.106:0/2202750063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 0x7fb19c1031b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:16.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.125+0000 7fb1a19d9700 1 -- 172.21.15.106:0/2202750063 >> 172.21.15.106:0/2202750063 conn(0x7fb19c077830 msgr2=0x7fb19c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:16.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.125+0000 7fb1a19d9700 1 -- 172.21.15.106:0/2202750063 shutdown_connections 2024-03-28T02:02:16.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.125+0000 7fb1a19d9700 1 -- 172.21.15.106:0/2202750063 wait complete. 2024-03-28T02:02:16.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.126+0000 7fb1a19d9700 1 Processor -- start 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.126+0000 7fb1a19d9700 1 -- start start 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.126+0000 7fb1a19d9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 0x7fb19c1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.126+0000 7fb1a19d9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19c10c1e0 con 0x7fb19c102dd0 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.126+0000 7fb19affd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 0x7fb19c1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.126+0000 7fb19affd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 0x7fb19c1976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:59232/0 (socket says 172.21.15.106:59232) 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.126+0000 7fb19affd700 1 -- 172.21.15.106:0/3295555909 learned_addr learned my addr 172.21.15.106:0/3295555909 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.127+0000 7fb19affd700 1 -- 172.21.15.106:0/3295555909 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c197c20 con 0x7fb19c102dd0 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.127+0000 7fb19affd700 1 --2- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 0x7fb19c1976e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb18c006010 tx=0x7fb18c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.127+0000 7fb1a09d7700 1 -- 172.21.15.106:0/3295555909 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c026070 con 0x7fb19c102dd0 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.127+0000 7fb1a09d7700 1 -- 172.21.15.106:0/3295555909 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb18c00bc30 con 0x7fb19c102dd0 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.127+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19c197eb0 con 0x7fb19c102dd0 2024-03-28T02:02:16.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.127+0000 7fb1a09d7700 1 -- 172.21.15.106:0/3295555909 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c003ab0 con 0x7fb19c102dd0 2024-03-28T02:02:16.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.127+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19c198310 con 0x7fb19c102dd0 2024-03-28T02:02:16.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.128+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19c069da0 con 0x7fb19c102dd0 2024-03-28T02:02:16.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.128+0000 7fb1a09d7700 1 -- 172.21.15.106:0/3295555909 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fb18c029070 con 0x7fb19c102dd0 2024-03-28T02:02:16.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.128+0000 7fb1a09d7700 1 --2- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb184038fc0 0x7fb18403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:16.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.128+0000 7fb1a09d7700 1 -- 172.21.15.106:0/3295555909 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb18c01e070 con 0x7fb19c102dd0 2024-03-28T02:02:16.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.128+0000 7fb19a7fc700 1 --2- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb184038fc0 0x7fb18403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:16.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.129+0000 7fb19a7fc700 1 --2- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb184038fc0 0x7fb18403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb190006fd0 tx=0x7fb190006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:16.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.133+0000 7fb1a09d7700 1 -- 172.21.15.106:0/3295555909 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb18c013070 con 0x7fb19c102dd0 2024-03-28T02:02:16.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.337+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb19c198a40 con 0x7fb184038fc0 2024-03-28T02:02:16.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.338+0000 7fb1a09d7700 1 -- 172.21.15.106:0/3295555909 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7fb19c198a40 con 0x7fb184038fc0 2024-03-28T02:02:16.333 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:02:16.333 INFO:teuthology.orchestra.run.smithi106.stdout:[{"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}, {"addr": "172.21.15.178", "hostname": "smithi178", "labels": [], "status": ""}] 2024-03-28T02:02:16.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb184038fc0 msgr2=0x7fb18403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:16.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 --2- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb184038fc0 0x7fb18403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb190006fd0 tx=0x7fb190006e40 comp rx=0 tx=0).stop 2024-03-28T02:02:16.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 msgr2=0x7fb19c1976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:16.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 --2- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 0x7fb19c1976e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb18c006010 tx=0x7fb18c005050 comp rx=0 tx=0).stop 2024-03-28T02:02:16.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 shutdown_connections 2024-03-28T02:02:16.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 --2- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb184038fc0 0x7fb18403b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:16.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 --2- 172.21.15.106:0/3295555909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19c102dd0 0x7fb19c1976e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:16.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 >> 172.21.15.106:0/3295555909 conn(0x7fb19c077830 msgr2=0x7fb19c103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:16.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 shutdown_connections 2024-03-28T02:02:16.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:16.341+0000 7fb1a19d9700 1 -- 172.21.15.106:0/3295555909 wait complete. 2024-03-28T02:02:16.909 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T02:02:16.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T02:02:17.142 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:02:17.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:17 smithi106 ceph-mon[31965]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:17.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:17 smithi106 ceph-mon[31965]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:02:17.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:18.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.782+0000 7f631f5e7700 1 -- 172.21.15.106:0/4009172888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 msgr2=0x7f6318106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:18.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.782+0000 7f631f5e7700 1 --2- 172.21.15.106:0/4009172888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 0x7f6318106e50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6314009910 tx=0x7f6314009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:18.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.782+0000 7f631f5e7700 1 -- 172.21.15.106:0/4009172888 shutdown_connections 2024-03-28T02:02:18.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.782+0000 7f631f5e7700 1 --2- 172.21.15.106:0/4009172888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 0x7f6318106e50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.782+0000 7f631f5e7700 1 -- 172.21.15.106:0/4009172888 >> 172.21.15.106:0/4009172888 conn(0x7f63180fc5c0 msgr2=0x7f63180fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.783+0000 7f631f5e7700 1 -- 172.21.15.106:0/4009172888 shutdown_connections 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.783+0000 7f631f5e7700 1 -- 172.21.15.106:0/4009172888 wait complete. 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.783+0000 7f631f5e7700 1 Processor -- start 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.783+0000 7f631f5e7700 1 -- start start 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.783+0000 7f631f5e7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 0x7f631819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.783+0000 7f631f5e7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631810b0b0 con 0x7f6318106a70 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.783+0000 7f631d383700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 0x7f631819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.783+0000 7f631d383700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 0x7f631819f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39340/0 (socket says 172.21.15.106:39340) 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.783+0000 7f631d383700 1 -- 172.21.15.106:0/2709310941 learned_addr learned my addr 172.21.15.106:0/2709310941 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.784+0000 7f631d383700 1 -- 172.21.15.106:0/2709310941 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631819fb30 con 0x7f6318106a70 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.784+0000 7f631d383700 1 --2- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 0x7f631819f5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6314005850 tx=0x7f6314005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:18.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.784+0000 7f630e7fc700 1 -- 172.21.15.106:0/2709310941 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6314026070 con 0x7f6318106a70 2024-03-28T02:02:18.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.784+0000 7f630e7fc700 1 -- 172.21.15.106:0/2709310941 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f631400bbd0 con 0x7f6318106a70 2024-03-28T02:02:18.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.784+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631819fe20 con 0x7f6318106a70 2024-03-28T02:02:18.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.784+0000 7f630e7fc700 1 -- 172.21.15.106:0/2709310941 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6314003970 con 0x7f6318106a70 2024-03-28T02:02:18.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.784+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6318199890 con 0x7f6318106a70 2024-03-28T02:02:18.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.785+0000 7f630e7fc700 1 -- 172.21.15.106:0/2709310941 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f6314029070 con 0x7f6318106a70 2024-03-28T02:02:18.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.785+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63181021b0 con 0x7f6318106a70 2024-03-28T02:02:18.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.785+0000 7f630e7fc700 1 --2- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6304039020 0x7f630403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:18.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.785+0000 7f630e7fc700 1 -- 172.21.15.106:0/2709310941 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f631401e070 con 0x7f6318106a70 2024-03-28T02:02:18.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.785+0000 7f631cb82700 1 --2- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6304039020 0x7f630403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:18.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.785+0000 7f631cb82700 1 --2- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6304039020 0x7f630403b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6308006fd0 tx=0x7f6308006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:18.784 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.790+0000 7f630e7fc700 1 -- 172.21.15.106:0/2709310941 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6314013070 con 0x7f6318106a70 2024-03-28T02:02:18.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:18.988+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f631819abf0 con 0x7f6318106a70 2024-03-28T02:02:19.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.409+0000 7f630e7fc700 1 -- 172.21.15.106:0/2709310941 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f631400fe20 con 0x7f6318106a70 2024-03-28T02:02:19.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.412+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6304039020 msgr2=0x7f630403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:19.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.412+0000 7f631f5e7700 1 --2- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6304039020 0x7f630403b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6308006fd0 tx=0x7f6308006e40 comp rx=0 tx=0).stop 2024-03-28T02:02:19.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.412+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 msgr2=0x7f631819f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:19.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.412+0000 7f631f5e7700 1 --2- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 0x7f631819f5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6314005850 tx=0x7f6314005030 comp rx=0 tx=0).stop 2024-03-28T02:02:19.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.412+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 shutdown_connections 2024-03-28T02:02:19.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.412+0000 7f631f5e7700 1 --2- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6304039020 0x7f630403b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:19.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.412+0000 7f631f5e7700 1 --2- 172.21.15.106:0/2709310941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6318106a70 0x7f631819f5f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:19.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.412+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 >> 172.21.15.106:0/2709310941 conn(0x7f63180fc5c0 msgr2=0x7f6318069fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:19.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.413+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 shutdown_connections 2024-03-28T02:02:19.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:19.413+0000 7f631f5e7700 1 -- 172.21.15.106:0/2709310941 wait complete. 2024-03-28T02:02:19.408 INFO:teuthology.orchestra.run.smithi106.stderr:adjusted tunables profile to default 2024-03-28T02:02:19.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:19 smithi106 ceph-mon[31965]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:19.737 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:19 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2709310941' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T02:02:20.079 INFO:tasks.cephadm:Adding mon.a on smithi106 2024-03-28T02:02:20.080 INFO:tasks.cephadm:Adding mon.b on smithi153 2024-03-28T02:02:20.080 INFO:tasks.cephadm:Adding mon.c on smithi178 2024-03-28T02:02:20.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi106:172.21.15.106=a;smithi153:172.21.15.153=b;smithi178:172.21.15.178=c' 2024-03-28T02:02:20.312 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2709310941' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T02:02:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T02:02:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi178", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:02:20.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:20.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:20.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:20.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:20.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:21.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.819+0000 7f89a8709700 1 -- 172.21.15.178:0/50108714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 msgr2=0x7f89a0106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:21.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.819+0000 7f89a8709700 1 --2- 172.21.15.178:0/50108714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 0x7f89a0106e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f899c009910 tx=0x7f899c009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:21.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.819+0000 7f89a8709700 1 -- 172.21.15.178:0/50108714 shutdown_connections 2024-03-28T02:02:21.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.819+0000 7f89a8709700 1 --2- 172.21.15.178:0/50108714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 0x7f89a0106e50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:21.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.819+0000 7f89a8709700 1 -- 172.21.15.178:0/50108714 >> 172.21.15.178:0/50108714 conn(0x7f89a00fc5a0 msgr2=0x7f89a00fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:21.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.819+0000 7f89a8709700 1 -- 172.21.15.178:0/50108714 shutdown_connections 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.819+0000 7f89a8709700 1 -- 172.21.15.178:0/50108714 wait complete. 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.820+0000 7f89a8709700 1 Processor -- start 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.820+0000 7f89a8709700 1 -- start start 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.820+0000 7f89a8709700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 0x7f89a01984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.820+0000 7f89a8709700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a010b0b0 con 0x7f89a0106a70 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.820+0000 7f89a64a5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 0x7f89a01984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.820+0000 7f89a64a5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 0x7f89a01984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.178:45756/0 (socket says 172.21.15.178:45756) 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.820+0000 7f89a64a5700 1 -- 172.21.15.178:0/876762303 learned_addr learned my addr 172.21.15.178:0/876762303 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.821+0000 7f89a64a5700 1 -- 172.21.15.178:0/876762303 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a0198a20 con 0x7f89a0106a70 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.821+0000 7f89a64a5700 1 --2- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 0x7f89a01984e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f899c005850 tx=0x7f899c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.821+0000 7f89977fe700 1 -- 172.21.15.178:0/876762303 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f899c026070 con 0x7f89a0106a70 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.821+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a0198d10 con 0x7f89a0106a70 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.821+0000 7f89977fe700 1 -- 172.21.15.178:0/876762303 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f899c00bbd0 con 0x7f89a0106a70 2024-03-28T02:02:21.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.821+0000 7f89977fe700 1 -- 172.21.15.178:0/876762303 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f899c003970 con 0x7f89a0106a70 2024-03-28T02:02:21.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.821+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a01991d0 con 0x7f89a0106a70 2024-03-28T02:02:21.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.822+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a01021b0 con 0x7f89a0106a70 2024-03-28T02:02:21.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.822+0000 7f89977fe700 1 -- 172.21.15.178:0/876762303 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f899c029070 con 0x7f89a0106a70 2024-03-28T02:02:21.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.822+0000 7f89977fe700 1 --2- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f898c038fc0 0x7f898c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:21.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.822+0000 7f89977fe700 1 -- 172.21.15.178:0/876762303 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f899c01e070 con 0x7f89a0106a70 2024-03-28T02:02:21.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.822+0000 7f89a5ca4700 1 --2- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f898c038fc0 0x7f898c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:21.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.823+0000 7f89a5ca4700 1 --2- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f898c038fc0 0x7f898c03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8990006fd0 tx=0x7f8990006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:21.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:21.827+0000 7f89977fe700 1 -- 172.21.15.178:0/876762303 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f899c013070 con 0x7f89a0106a70 2024-03-28T02:02:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:21 smithi106 ceph-mon[31965]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-28T02:02:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:21 smithi106 ceph-mon[31965]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:21 smithi106 ceph-mon[31965]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:02:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:21 smithi106 ceph-mon[31965]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:02:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:21 smithi106 ceph-mon[31965]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:22.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.032+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi106:172.21.15.106=a;smithi153:172.21.15.153=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f89a006a5b0 con 0x7f898c038fc0 2024-03-28T02:02:22.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.038+0000 7f89977fe700 1 -- 172.21.15.178:0/876762303 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f89a006a5b0 con 0x7f898c038fc0 2024-03-28T02:02:22.033 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mon update... 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.040+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f898c038fc0 msgr2=0x7f898c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.040+0000 7f89a8709700 1 --2- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f898c038fc0 0x7f898c03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8990006fd0 tx=0x7f8990006e40 comp rx=0 tx=0).stop 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.040+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 msgr2=0x7f89a01984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.040+0000 7f89a8709700 1 --2- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 0x7f89a01984e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f899c005850 tx=0x7f899c005030 comp rx=0 tx=0).stop 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.041+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 shutdown_connections 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.041+0000 7f89a8709700 1 --2- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f898c038fc0 0x7f898c03b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.041+0000 7f89a8709700 1 --2- 172.21.15.178:0/876762303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89a0106a70 0x7f89a01984e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.041+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 >> 172.21.15.178:0/876762303 conn(0x7f89a00fc5a0 msgr2=0x7f89a00fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.041+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 shutdown_connections 2024-03-28T02:02:22.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:22.041+0000 7f89a8709700 1 -- 172.21.15.178:0/876762303 wait complete. 2024-03-28T02:02:22.642 DEBUG:teuthology.orchestra.run.smithi153:mon.b> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T02:02:22.645 DEBUG:teuthology.orchestra.run.smithi178:mon.c> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T02:02:22.647 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T02:02:22.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T02:02:23.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi106:172.21.15.106=a;smithi153:172.21.15.153=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:23.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: Saving service mon spec with placement smithi106:172.21.15.106=a;smithi153:172.21.15.153=b;smithi178:172.21.15.178=c;count:3 2024-03-28T02:02:23.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:23.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:23.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:23.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:23.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:23.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:23.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:23.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: Deploying daemon mon.c on smithi178 2024-03-28T02:02:23.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:23 smithi106 ceph-mon[31965]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:23.701 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:25.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:25 smithi106 ceph-mon[31965]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.121+0000 7f6574865700 1 -- 172.21.15.178:0/903999179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 msgr2=0x7f65701021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.121+0000 7f6574865700 1 --2- 172.21.15.178:0/903999179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 0x7f65701021b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6560009910 tx=0x7f6560009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.121+0000 7f6574865700 1 -- 172.21.15.178:0/903999179 shutdown_connections 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.121+0000 7f6574865700 1 --2- 172.21.15.178:0/903999179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 0x7f65701021b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.121+0000 7f6574865700 1 -- 172.21.15.178:0/903999179 >> 172.21.15.178:0/903999179 conn(0x7f65700fd660 msgr2=0x7f65700ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.121+0000 7f6574865700 1 -- 172.21.15.178:0/903999179 shutdown_connections 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.121+0000 7f6574865700 1 -- 172.21.15.178:0/903999179 wait complete. 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.122+0000 7f6574865700 1 Processor -- start 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.122+0000 7f6574865700 1 -- start start 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.122+0000 7f6574865700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 0x7f6570074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.122+0000 7f6574865700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657010c1a0 con 0x7f6570101dd0 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.122+0000 7f656e59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 0x7f6570074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:26.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.122+0000 7f656e59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 0x7f6570074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.178:45788/0 (socket says 172.21.15.178:45788) 2024-03-28T02:02:26.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.122+0000 7f656e59c700 1 -- 172.21.15.178:0/3779480469 learned_addr learned my addr 172.21.15.178:0/3779480469 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:02:26.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.122+0000 7f656e59c700 1 -- 172.21.15.178:0/3779480469 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6570076380 con 0x7f6570101dd0 2024-03-28T02:02:26.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.122+0000 7f656e59c700 1 --2- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 0x7f6570074490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6560005850 tx=0x7f6560005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:26.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.123+0000 7f655f7fe700 1 -- 172.21.15.178:0/3779480469 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6560026070 con 0x7f6570101dd0 2024-03-28T02:02:26.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.123+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6570074a30 con 0x7f6570101dd0 2024-03-28T02:02:26.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.123+0000 7f655f7fe700 1 -- 172.21.15.178:0/3779480469 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f656000bc30 con 0x7f6570101dd0 2024-03-28T02:02:26.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.123+0000 7f655f7fe700 1 -- 172.21.15.178:0/3779480469 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6560003a20 con 0x7f6570101dd0 2024-03-28T02:02:26.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.123+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6570074ef0 con 0x7f6570101dd0 2024-03-28T02:02:26.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.124+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65701032b0 con 0x7f6570101dd0 2024-03-28T02:02:26.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.124+0000 7f655f7fe700 1 -- 172.21.15.178:0/3779480469 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f6560029070 con 0x7f6570101dd0 2024-03-28T02:02:26.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.124+0000 7f655f7fe700 1 --2- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6558039020 0x7f655803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:26.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.124+0000 7f655f7fe700 1 -- 172.21.15.178:0/3779480469 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f656001e070 con 0x7f6570101dd0 2024-03-28T02:02:26.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.124+0000 7f656dd9b700 1 --2- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6558039020 0x7f655803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:26.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.125+0000 7f656dd9b700 1 --2- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6558039020 0x7f655803b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6564006fd0 tx=0x7f6564006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:26.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.129+0000 7f655f7fe700 1 -- 172.21.15.178:0/3779480469 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6560013070 con 0x7f6570101dd0 2024-03-28T02:02:26.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.423+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65700759c0 con 0x7f6570101dd0 2024-03-28T02:02:26.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.424+0000 7f655f7fe700 1 -- 172.21.15.178:0/3779480469 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7f6560030090 con 0x7f6570101dd0 2024-03-28T02:02:26.418 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T02:02:26.419 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","modified":"2024-03-28T02:00:36.905339Z","created":"2024-03-28T02:00:36.905339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T02:02:26.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6558039020 msgr2=0x7f655803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:26.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 --2- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6558039020 0x7f655803b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6564006fd0 tx=0x7f6564006e40 comp rx=0 tx=0).stop 2024-03-28T02:02:26.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 msgr2=0x7f6570074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:26.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 --2- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 0x7f6570074490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6560005850 tx=0x7f6560005030 comp rx=0 tx=0).stop 2024-03-28T02:02:26.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 shutdown_connections 2024-03-28T02:02:26.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 --2- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6558039020 0x7f655803b4e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:26.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 --2- 172.21.15.178:0/3779480469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6570101dd0 0x7f6570074490 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:26.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 >> 172.21.15.178:0/3779480469 conn(0x7f65700fd660 msgr2=0x7f65701037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:26.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 shutdown_connections 2024-03-28T02:02:26.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:26.426+0000 7f6574865700 1 -- 172.21.15.178:0/3779480469 wait complete. 2024-03-28T02:02:26.421 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-03-28T02:02:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:27 smithi106 ceph-mon[31965]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:27 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/3779480469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T02:02:27.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:27 smithi178 systemd[1]: Started Ceph mon.c for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:02:27.996 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T02:02:27.997 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T02:02:28.230 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:02:32.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.604+0000 7fd179c9a700 1 -- 172.21.15.178:0/990339076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1600035c0 msgr2=0x7fd160005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:32.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.605+0000 7fd179c9a700 1 --2- 172.21.15.178:0/990339076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1600035c0 0x7fd160005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd174101400 tx=0x7fd16400feb0 comp rx=0 tx=0).stop 2024-03-28T02:02:32.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.605+0000 7fd179c9a700 1 -- 172.21.15.178:0/990339076 shutdown_connections 2024-03-28T02:02:32.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.605+0000 7fd179c9a700 1 --2- 172.21.15.178:0/990339076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1600035c0 0x7fd160005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:32.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.605+0000 7fd179c9a700 1 -- 172.21.15.178:0/990339076 >> 172.21.15.178:0/990339076 conn(0x7fd174077280 msgr2=0x7fd174077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:32.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.605+0000 7fd179c9a700 1 -- 172.21.15.178:0/990339076 shutdown_connections 2024-03-28T02:02:32.600 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.605+0000 7fd179c9a700 1 -- 172.21.15.178:0/990339076 wait complete. 2024-03-28T02:02:32.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.605+0000 7fd179c9a700 1 Processor -- start 2024-03-28T02:02:32.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.605+0000 7fd179c9a700 1 -- start start 2024-03-28T02:02:32.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd179c9a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 0x7fd17419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd179c9a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd17419fc10 0x7fd174199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd179c9a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd174109d20 con 0x7fd17419fc10 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd179c9a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd174109ba0 con 0x7fd1600035c0 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd1737fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 0x7fd17419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd1737fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 0x7fd17419f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:52842/0 (socket says 172.21.15.178:52842) 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd1737fe700 1 -- 172.21.15.178:0/1724136458 learned_addr learned my addr 172.21.15.178:0/1724136458 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd16affd700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd17419fc10 0x7fd174199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd1737fe700 1 -- 172.21.15.178:0/1724136458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 msgr2=0x7fd17419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd1737fe700 1 -- 172.21.15.178:0/1724136458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 msgr2=0x7fd17419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd1737fe700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 0x7fd17419f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd1737fe700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 0x7fd17419f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd16affd700 1 -- 172.21.15.178:0/1724136458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 msgr2=0x7fd17419f6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T02:02:32.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd16affd700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 0x7fd17419f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:32.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.606+0000 7fd16affd700 1 -- 172.21.15.178:0/1724136458 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1741a0250 con 0x7fd17419fc10 2024-03-28T02:02:32.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.607+0000 7fd16affd700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd17419fc10 0x7fd174199850 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00b4d0 tx=0x7fd15c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:32.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.607+0000 7fd1717fa700 1 -- 172.21.15.178:0/1724136458 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd15c011630 con 0x7fd17419fc10 2024-03-28T02:02:32.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.607+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd174199df0 con 0x7fd17419fc10 2024-03-28T02:02:32.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.607+0000 7fd1717fa700 1 -- 172.21.15.178:0/1724136458 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd15c011cb0 con 0x7fd17419fc10 2024-03-28T02:02:32.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.607+0000 7fd1717fa700 1 -- 172.21.15.178:0/1724136458 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd15c010e60 con 0x7fd17419fc10 2024-03-28T02:02:32.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.607+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17419a350 con 0x7fd17419fc10 2024-03-28T02:02:32.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.608+0000 7fd1717fa700 1 -- 172.21.15.178:0/1724136458 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fd15c018020 con 0x7fd17419fc10 2024-03-28T02:02:32.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.608+0000 7fd1717fa700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd154038d10 0x7fd15403b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:32.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.608+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17406a0d0 con 0x7fd17419fc10 2024-03-28T02:02:32.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.608+0000 7fd1717fa700 1 -- 172.21.15.178:0/1724136458 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd15c04ba90 con 0x7fd17419fc10 2024-03-28T02:02:32.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.608+0000 7fd1737fe700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd154038d10 0x7fd15403b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:32.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.609+0000 7fd1737fe700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd154038d10 0x7fd15403b1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd16400fa60 tx=0x7fd164005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:32.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.613+0000 7fd1717fa700 1 -- 172.21.15.178:0/1724136458 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd15c01f020 con 0x7fd17419fc10 2024-03-28T02:02:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:02:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: mon.a calling monitor election 2024-03-28T02:02:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: mon.c calling monitor election 2024-03-28T02:02:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: monmap e2: 2 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: fsmap 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: mgrmap e13: x(active, since 50s) 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: overall HEALTH_OK 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:32.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: mon.a calling monitor election 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: mon.c calling monitor election 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:32.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T02:02:32.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: monmap e2: 2 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T02:02:32.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: fsmap 2024-03-28T02:02:32.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T02:02:32.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: mgrmap e13: x(active, since 50s) 2024-03-28T02:02:32.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: overall HEALTH_OK 2024-03-28T02:02:32.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:32.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:32.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.887+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd17419a6c0 con 0x7fd17419fc10 2024-03-28T02:02:32.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.887+0000 7fd1717fa700 1 -- 172.21.15.178:0/1724136458 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7fd15c018310 con 0x7fd17419fc10 2024-03-28T02:02:32.881 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T02:02:32.882 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":2,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","modified":"2024-03-28T02:02:27.409326Z","created":"2024-03-28T02:00:36.905339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:3300","nonce":0},{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.889+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd154038d10 msgr2=0x7fd15403b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.889+0000 7fd179c9a700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd154038d10 0x7fd15403b1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd16400fa60 tx=0x7fd164005a90 comp rx=0 tx=0).stop 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.889+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd17419fc10 msgr2=0x7fd174199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.889+0000 7fd179c9a700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd17419fc10 0x7fd174199850 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00b4d0 tx=0x7fd15c00b890 comp rx=0 tx=0).stop 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.890+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 shutdown_connections 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.890+0000 7fd179c9a700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1600035c0 0x7fd17419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.890+0000 7fd179c9a700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd154038d10 0x7fd15403b1d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.890+0000 7fd179c9a700 1 --2- 172.21.15.178:0/1724136458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd17419fc10 0x7fd174199850 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.890+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 >> 172.21.15.178:0/1724136458 conn(0x7fd174077280 msgr2=0x7fd174102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:32.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.890+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 shutdown_connections 2024-03-28T02:02:32.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:32.890+0000 7fd179c9a700 1 -- 172.21.15.178:0/1724136458 wait complete. 2024-03-28T02:02:32.885 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 2 2024-03-28T02:02:33.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:33 smithi106 ceph-mon[31965]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:33.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:33.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:33.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:33.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:33 smithi106 ceph-mon[31965]: Deploying daemon mon.b on smithi153 2024-03-28T02:02:33.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:33 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/1724136458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T02:02:33.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:33 smithi178 ceph-mon[36521]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:33 smithi178 ceph-mon[36521]: Deploying daemon mon.b on smithi153 2024-03-28T02:02:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:33 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/1724136458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T02:02:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:34.478 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T02:02:34.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T02:02:34.723 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:02:34.834 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:02:34 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:02:34.409+0000 7efc1195f700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T02:02:35.798 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:35 smithi178 ceph-mon[36521]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:35 smithi106 ceph-mon[31965]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.148+0000 7f53b124a700 1 -- 172.21.15.178:0/1608675818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac102e70 msgr2=0x7f53ac103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:36.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.148+0000 7f53b124a700 1 --2- 172.21.15.178:0/1608675818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac102e70 0x7f53ac103250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009910 tx=0x7f53a0009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:36.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.148+0000 7f53b124a700 1 -- 172.21.15.178:0/1608675818 shutdown_connections 2024-03-28T02:02:36.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.148+0000 7f53b124a700 1 --2- 172.21.15.178:0/1608675818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac102e70 0x7f53ac103250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:36.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.148+0000 7f53b124a700 1 -- 172.21.15.178:0/1608675818 >> 172.21.15.178:0/1608675818 conn(0x7f53ac0fc640 msgr2=0x7f53ac0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:36.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.148+0000 7f53b124a700 1 -- 172.21.15.178:0/1608675818 shutdown_connections 2024-03-28T02:02:36.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.148+0000 7f53b124a700 1 -- 172.21.15.178:0/1608675818 wait complete. 2024-03-28T02:02:36.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.149+0000 7f53b124a700 1 Processor -- start 2024-03-28T02:02:36.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.149+0000 7f53b124a700 1 -- start start 2024-03-28T02:02:36.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.149+0000 7f53b124a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 0x7f53ac19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:36.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.149+0000 7f53b124a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac19db90 0x7f53ac1977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:36.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.149+0000 7f53b124a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac10c2f0 con 0x7f53ac19db90 2024-03-28T02:02:36.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.149+0000 7f53b124a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac10c170 con 0x7f53ac102e70 2024-03-28T02:02:36.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aad9d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 0x7f53ac19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:36.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aad9d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 0x7f53ac19d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:52848/0 (socket says 172.21.15.178:52848) 2024-03-28T02:02:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aad9d700 1 -- 172.21.15.178:0/4112118279 learned_addr learned my addr 172.21.15.178:0/4112118279 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:02:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aa59c700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac19db90 0x7f53ac1977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aad9d700 1 -- 172.21.15.178:0/4112118279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 msgr2=0x7f53ac19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T02:02:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aad9d700 1 -- 172.21.15.178:0/4112118279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 msgr2=0x7f53ac19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T02:02:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aad9d700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 0x7f53ac19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T02:02:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aad9d700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 0x7f53ac19d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T02:02:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aa59c700 1 -- 172.21.15.178:0/4112118279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 msgr2=0x7f53ac19d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T02:02:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aa59c700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 0x7f53ac19d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aa59c700 1 -- 172.21.15.178:0/4112118279 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac19e1d0 con 0x7f53ac19db90 2024-03-28T02:02:36.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.150+0000 7f53aa59c700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac19db90 0x7f53ac1977d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f539c00b4d0 tx=0x7f539c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:36.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.151+0000 7f5393fff700 1 -- 172.21.15.178:0/4112118279 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f539c011630 con 0x7f53ac19db90 2024-03-28T02:02:36.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.151+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ac197d70 con 0x7f53ac19db90 2024-03-28T02:02:36.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.151+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ac1982d0 con 0x7f53ac19db90 2024-03-28T02:02:36.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.151+0000 7f5393fff700 1 -- 172.21.15.178:0/4112118279 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f539c011cb0 con 0x7f53ac19db90 2024-03-28T02:02:36.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.151+0000 7f5393fff700 1 -- 172.21.15.178:0/4112118279 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f539c0193b0 con 0x7f53ac19db90 2024-03-28T02:02:36.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.152+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ac06a360 con 0x7f53ac19db90 2024-03-28T02:02:36.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.152+0000 7f5393fff700 1 -- 172.21.15.178:0/4112118279 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f539c018020 con 0x7f53ac19db90 2024-03-28T02:02:36.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.152+0000 7f5393fff700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5394038d60 0x7f539403b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:36.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.152+0000 7f5393fff700 1 -- 172.21.15.178:0/4112118279 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f539c04ba10 con 0x7f53ac19db90 2024-03-28T02:02:36.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.152+0000 7f53aad9d700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5394038d60 0x7f539403b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:36.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.152+0000 7f53aad9d700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5394038d60 0x7f539403b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f53a0000c00 tx=0x7f53a000b580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:36.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.157+0000 7f5393fff700 1 -- 172.21.15.178:0/4112118279 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f539c01f020 con 0x7f53ac19db90 2024-03-28T02:02:36.319 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).mds e1 new map 2024-03-28T02:02:36.319 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-28T02:02:36.319 INFO:journalctl@ceph.mon.b.smithi153.stdout: e1 2024-03-28T02:02:36.319 INFO:journalctl@ceph.mon.b.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T02:02:36.319 INFO:journalctl@ceph.mon.b.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T02:02:36.319 INFO:journalctl@ceph.mon.b.smithi153.stdout: legacy client fscid: -1 2024-03-28T02:02:36.320 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-28T02:02:36.320 INFO:journalctl@ceph.mon.b.smithi153.stdout: No filesystems configured 2024-03-28T02:02:36.320 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T02:02:36.320 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T02:02:36.320 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T02:02:36.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.331+0000 7f5393fff700 1 -- 172.21.15.178:0/4112118279 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c0174a0 con 0x7f53ac19db90 2024-03-28T02:02:36.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:36.437+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53ac198640 con 0x7f53ac19db90 2024-03-28T02:02:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T02:02:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T02:02:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T02:02:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T02:02:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T02:02:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:36.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Deploying cephadm binary to smithi178 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Added host smithi178 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2709310941' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2709310941' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi178", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi106:172.21.15.106=a;smithi153:172.21.15.153=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Saving service mon spec with placement smithi106:172.21.15.106=a;smithi153:172.21.15.153=b;smithi178:172.21.15.178=c;count:3 2024-03-28T02:02:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Deploying daemon mon.c on smithi178 2024-03-28T02:02:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/3779480469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.a calling monitor election 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.c calling monitor election 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: monmap e2: 2 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: fsmap 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mgrmap e13: x(active, since 50s) 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: overall HEALTH_OK 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: Deploying daemon mon.b on smithi153 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/1724136458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T02:02:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:36.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:36.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 ceph-mon[36228]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T02:02:36.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 bash[36209]: 03b91e3f4431c09af177c044a7f03176d5fe1da3ddafb28d519bf24ec422f2e8 2024-03-28T02:02:36.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:36 smithi153 systemd[1]: Started Ceph mon.b for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:02:41.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.377+0000 7f5393fff700 1 -- 172.21.15.178:0/4112118279 <== mon.0 v2:172.21.15.106:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1289 (secure 0 0 0) 0x7f539c018550 con 0x7f53ac19db90 2024-03-28T02:02:41.371 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-28T02:02:41.371 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":3,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","modified":"2024-03-28T02:02:36.331166Z","created":"2024-03-28T02:00:36.905339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:3300","nonce":0},{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-28T02:02:41.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5394038d60 msgr2=0x7f539403b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:41.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5394038d60 0x7f539403b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f53a0000c00 tx=0x7f53a000b580 comp rx=0 tx=0).stop 2024-03-28T02:02:41.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac19db90 msgr2=0x7f53ac1977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:41.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac19db90 0x7f53ac1977d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f539c00b4d0 tx=0x7f539c00b890 comp rx=0 tx=0).stop 2024-03-28T02:02:41.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 shutdown_connections 2024-03-28T02:02:41.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53ac102e70 0x7f53ac19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:41.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5394038d60 0x7f539403b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:41.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 --2- 172.21.15.178:0/4112118279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac19db90 0x7f53ac1977d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:41.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 >> 172.21.15.178:0/4112118279 conn(0x7f53ac0fc640 msgr2=0x7f53ac103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:41.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 shutdown_connections 2024-03-28T02:02:41.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:41.380+0000 7f53b124a700 1 -- 172.21.15.178:0/4112118279 wait complete. 2024-03-28T02:02:41.375 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 3 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: mon.a calling monitor election 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: mon.c calling monitor election 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/4112118279' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: mon.b calling monitor election 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: monmap e3: 3 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: fsmap 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: mgrmap e13: x(active, since 59s) 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: overall HEALTH_OK 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: mon.a calling monitor election 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: mon.c calling monitor election 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/4112118279' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: mon.b calling monitor election 2024-03-28T02:02:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: monmap e3: 3 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: fsmap 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: mgrmap e13: x(active, since 59s) 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: overall HEALTH_OK 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: mon.a calling monitor election 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: mon.c calling monitor election 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/4112118279' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: mon.b calling monitor election 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: monmap e3: 3 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: fsmap 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: mgrmap e13: x(active, since 59s) 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: overall HEALTH_OK 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:41.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:42.056 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T02:02:42.056 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T02:02:42.299 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:02:42.621 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:42.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:42.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:42.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:42.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:42.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:42.739 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:42.739 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:02:42.739 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:43.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:43.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:43.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:43.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:43.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:43.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:43.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.025+0000 7f4afa65b700 1 -- 172.21.15.106:0/3146765142 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 msgr2=0x7f4af4102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:44.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.025+0000 7f4afa65b700 1 --2- 172.21.15.106:0/3146765142 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 0x7f4af4102190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009910 tx=0x7f4ae4009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:44.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.025+0000 7f4afa65b700 1 -- 172.21.15.106:0/3146765142 shutdown_connections 2024-03-28T02:02:44.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.025+0000 7f4afa65b700 1 --2- 172.21.15.106:0/3146765142 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 0x7f4af4102190 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:44.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.025+0000 7f4afa65b700 1 -- 172.21.15.106:0/3146765142 >> 172.21.15.106:0/3146765142 conn(0x7f4af40fd660 msgr2=0x7f4af40ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:44.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.025+0000 7f4afa65b700 1 -- 172.21.15.106:0/3146765142 shutdown_connections 2024-03-28T02:02:44.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.025+0000 7f4afa65b700 1 -- 172.21.15.106:0/3146765142 wait complete. 2024-03-28T02:02:44.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4afa65b700 1 Processor -- start 2024-03-28T02:02:44.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4afa65b700 1 -- start start 2024-03-28T02:02:44.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4afa65b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 0x7f4af4074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4afa65b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4af40749d0 0x7f4af4074e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4afa65b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4af41a3ac0 0x7f4af41a5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4afa65b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af410c180 con 0x7f4af4101db0 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4afa65b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af410c000 con 0x7f4af40749d0 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4afa65b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af410c300 con 0x7f4af41a3ac0 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4af3fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 0x7f4af4074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4af3fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 0x7f4af4074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:43964/0 (socket says 172.21.15.106:43964) 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.026+0000 7f4af3fff700 1 -- 172.21.15.106:0/2036172935 learned_addr learned my addr 172.21.15.106:0/2036172935 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4af3fff700 1 -- 172.21.15.106:0/2036172935 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4af41a3ac0 msgr2=0x7f4af41a5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4af3fff700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4af41a3ac0 0x7f4af41a5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4af3fff700 1 -- 172.21.15.106:0/2036172935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4af40749d0 msgr2=0x7f4af4074e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4af3fff700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4af40749d0 0x7f4af4074e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4af3fff700 1 -- 172.21.15.106:0/2036172935 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af41a65e0 con 0x7f4af4101db0 2024-03-28T02:02:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4af3fff700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 0x7f4af4074490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4005850 tx=0x7f4ae4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:44.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4af17fa700 1 -- 172.21.15.106:0/2036172935 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4026070 con 0x7f4af4101db0 2024-03-28T02:02:44.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4af17fa700 1 -- 172.21.15.106:0/2036172935 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ae400b6f0 con 0x7f4af4101db0 2024-03-28T02:02:44.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af41a68d0 con 0x7f4af4101db0 2024-03-28T02:02:44.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4af17fa700 1 -- 172.21.15.106:0/2036172935 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae40039e0 con 0x7f4af4101db0 2024-03-28T02:02:44.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.027+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af41a6e10 con 0x7f4af4101db0 2024-03-28T02:02:44.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.028+0000 7f4af17fa700 1 -- 172.21.15.106:0/2036172935 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f4ae4029070 con 0x7f4af4101db0 2024-03-28T02:02:44.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.028+0000 7f4af17fa700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4adc039450 0x7f4adc03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:44.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.028+0000 7f4af17fa700 1 -- 172.21.15.106:0/2036172935 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4ae404cce0 con 0x7f4af4101db0 2024-03-28T02:02:44.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.028+0000 7f4af37fe700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4adc039450 0x7f4adc03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:44.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.028+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af40637c0 con 0x7f4af4101db0 2024-03-28T02:02:44.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.029+0000 7f4af37fe700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4adc039450 0x7f4adc03b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80098e0 tx=0x7f4ae8006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:44.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.033+0000 7f4af17fa700 1 -- 172.21.15.106:0/2036172935 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ae4030070 con 0x7f4af4101db0 2024-03-28T02:02:44.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.238+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f4af41a7040 con 0x7f4af4101db0 2024-03-28T02:02:44.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.238+0000 7f4af17fa700 1 -- 172.21.15.106:0/2036172935 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f4ae4013070 con 0x7f4af4101db0 2024-03-28T02:02:44.233 INFO:teuthology.orchestra.run.smithi106.stdout:# minimal ceph.conf for d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:02:44.233 INFO:teuthology.orchestra.run.smithi106.stdout:[global] 2024-03-28T02:02:44.233 INFO:teuthology.orchestra.run.smithi106.stdout: fsid = d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:02:44.233 INFO:teuthology.orchestra.run.smithi106.stdout: mon_host = [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-03-28T02:02:44.236 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4adc039450 msgr2=0x7f4adc03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:44.236 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4adc039450 0x7f4adc03b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80098e0 tx=0x7f4ae8006dd0 comp rx=0 tx=0).stop 2024-03-28T02:02:44.236 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 msgr2=0x7f4af4074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:44.236 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 0x7f4af4074490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4005850 tx=0x7f4ae4005030 comp rx=0 tx=0).stop 2024-03-28T02:02:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 shutdown_connections 2024-03-28T02:02:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4af41a3ac0 0x7f4af41a5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4adc039450 0x7f4adc03b910 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af4101db0 0x7f4af4074490 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 --2- 172.21.15.106:0/2036172935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4af40749d0 0x7f4af4074e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 >> 172.21.15.106:0/2036172935 conn(0x7f4af40fd660 msgr2=0x7f4af4103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 shutdown_connections 2024-03-28T02:02:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:44.242+0000 7f4afa65b700 1 -- 172.21.15.106:0/2036172935 wait complete. 2024-03-28T02:02:44.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-28T02:02:44.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: Reconfiguring daemon mon.a on smithi106 2024-03-28T02:02:44.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:44 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2036172935' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T02:02:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:44.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T02:02:44.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:44.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: Reconfiguring daemon mon.a on smithi106 2024-03-28T02:02:44.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:44 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2036172935' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.774 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:44.775 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T02:02:44.775 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:44.775 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T02:02:44.775 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:44.775 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: Reconfiguring daemon mon.a on smithi106 2024-03-28T02:02:44.775 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:44 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2036172935' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:45 smithi106 ceph-mon[31965]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T02:02:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:45.609 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T02:02:45.609 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:02:45.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T02:02:45.637 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:02:45.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:02:45.705 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T02:02:45.705 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T02:02:45.734 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T02:02:45.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:02:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:45 smithi153 ceph-mon[36228]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T02:02:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:45.801 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T02:02:45.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T02:02:45.826 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:45 smithi178 ceph-mon[36521]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:45.827 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:45.827 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:45.827 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:45.827 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T02:02:45.827 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:45.831 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T02:02:45.831 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:02:45.898 INFO:tasks.cephadm:Adding mgr.x on smithi106 2024-03-28T02:02:45.898 INFO:tasks.cephadm:Adding mgr.y on smithi153 2024-03-28T02:02:45.898 INFO:tasks.cephadm:Adding mgr.z on smithi178 2024-03-28T02:02:45.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi106=x;smithi153=y;smithi178=z' 2024-03-28T02:02:46.167 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:02:46.554 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:46 smithi178 ceph-mon[36521]: Reconfiguring mon.b (monmap changed)... 2024-03-28T02:02:46.555 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:46 smithi178 ceph-mon[36521]: Reconfiguring daemon mon.b on smithi153 2024-03-28T02:02:46.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:46 smithi153 ceph-mon[36228]: Reconfiguring mon.b (monmap changed)... 2024-03-28T02:02:46.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:46 smithi153 ceph-mon[36228]: Reconfiguring daemon mon.b on smithi153 2024-03-28T02:02:46.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:46 smithi106 ceph-mon[31965]: Reconfiguring mon.b (monmap changed)... 2024-03-28T02:02:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:46 smithi106 ceph-mon[31965]: Reconfiguring daemon mon.b on smithi153 2024-03-28T02:02:47.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.701+0000 7f3caaf0f700 1 -- 172.21.15.178:0/2187161030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 msgr2=0x7f3ca4102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:47.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.701+0000 7f3caaf0f700 1 --2- 172.21.15.178:0/2187161030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 0x7f3ca4102100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009910 tx=0x7f3c98009c20 comp rx=0 tx=0).stop 2024-03-28T02:02:47.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.701+0000 7f3caaf0f700 1 -- 172.21.15.178:0/2187161030 shutdown_connections 2024-03-28T02:02:47.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.701+0000 7f3caaf0f700 1 --2- 172.21.15.178:0/2187161030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 0x7f3ca4102100 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:47.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.701+0000 7f3caaf0f700 1 -- 172.21.15.178:0/2187161030 >> 172.21.15.178:0/2187161030 conn(0x7f3ca40fd280 msgr2=0x7f3ca40ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:47.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 -- 172.21.15.178:0/2187161030 shutdown_connections 2024-03-28T02:02:47.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 -- 172.21.15.178:0/2187161030 wait complete. 2024-03-28T02:02:47.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 Processor -- start 2024-03-28T02:02:47.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 -- start start 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 0x7f3ca4075ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ca4074580 0x7f3ca4074a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca4074f40 0x7f3ca41a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca4107d20 con 0x7f3ca4101ce0 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca4107ba0 con 0x7f3ca4074f40 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3caaf0f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca4107ea0 con 0x7f3ca4074580 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3ca8cab700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 0x7f3ca4075ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3ca3fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ca4074580 0x7f3ca4074a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3ca8cab700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 0x7f3ca4075ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.178:38808/0 (socket says 172.21.15.178:38808) 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.702+0000 7f3ca8cab700 1 -- 172.21.15.178:0/1071067718 learned_addr learned my addr 172.21.15.178:0/1071067718 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca94ac700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca4074f40 0x7f3ca41a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:47.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca8cab700 1 -- 172.21.15.178:0/1071067718 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ca4074580 msgr2=0x7f3ca4074a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca8cab700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ca4074580 0x7f3ca4074a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca8cab700 1 -- 172.21.15.178:0/1071067718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca4074f40 msgr2=0x7f3ca41a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca8cab700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca4074f40 0x7f3ca41a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca8cab700 1 -- 172.21.15.178:0/1071067718 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca41a1e40 con 0x7f3ca4101ce0 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca8cab700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 0x7f3ca4075ee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3c98000c00 tx=0x7f3c98005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca1ffb700 1 -- 172.21.15.178:0/1071067718 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98027070 con 0x7f3ca4101ce0 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca1ffb700 1 -- 172.21.15.178:0/1071067718 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c98022470 con 0x7f3ca4101ce0 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca41a20d0 con 0x7f3ca4101ce0 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3ca1ffb700 1 -- 172.21.15.178:0/1071067718 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800f680 con 0x7f3ca4101ce0 2024-03-28T02:02:47.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.703+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca41a2630 con 0x7f3ca4101ce0 2024-03-28T02:02:47.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.704+0000 7f3ca1ffb700 1 -- 172.21.15.178:0/1071067718 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f3c9802a070 con 0x7f3ca4101ce0 2024-03-28T02:02:47.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.704+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca40637c0 con 0x7f3ca4101ce0 2024-03-28T02:02:47.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.704+0000 7f3ca1ffb700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3c94039450 0x7f3c9403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:47.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.704+0000 7f3ca1ffb700 1 -- 172.21.15.178:0/1071067718 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3c9804d240 con 0x7f3ca4101ce0 2024-03-28T02:02:47.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.705+0000 7f3ca3fff700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3c94039450 0x7f3c9403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:47.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.705+0000 7f3ca3fff700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3c94039450 0x7f3c9403b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009ad0 tx=0x7f3c90009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:47.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.710+0000 7f3ca1ffb700 1 -- 172.21.15.178:0/1071067718 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3c98013070 con 0x7f3ca4101ce0 2024-03-28T02:02:47.808 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:47 smithi178 ceph-mon[36521]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:47.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:47.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:47.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:47.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T02:02:47.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:47.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:47 smithi106 ceph-mon[31965]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:47.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:47.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:47.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:47.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T02:02:47.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:47.916 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.921+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi106=x;smithi153=y;smithi178=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ca40ff140 con 0x7f3c94039450 2024-03-28T02:02:47.922 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.928+0000 7f3ca1ffb700 1 -- 172.21.15.178:0/1071067718 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3ca40ff140 con 0x7f3c94039450 2024-03-28T02:02:47.923 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mgr update... 2024-03-28T02:02:47.925 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3c94039450 msgr2=0x7f3c9403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:47.925 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3c94039450 0x7f3c9403b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009ad0 tx=0x7f3c90009210 comp rx=0 tx=0).stop 2024-03-28T02:02:47.925 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 msgr2=0x7f3ca4075ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:47.925 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 0x7f3ca4075ee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3c98000c00 tx=0x7f3c98005000 comp rx=0 tx=0).stop 2024-03-28T02:02:47.925 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 shutdown_connections 2024-03-28T02:02:47.925 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca4074f40 0x7f3ca41a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:47.925 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3c94039450 0x7f3c9403b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:47.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca4101ce0 0x7f3ca4075ee0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:47.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 --2- 172.21.15.178:0/1071067718 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ca4074580 0x7f3ca4074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:47.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 >> 172.21.15.178:0/1071067718 conn(0x7f3ca40fd280 msgr2=0x7f3ca40fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:47.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 shutdown_connections 2024-03-28T02:02:47.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:02:47.930+0000 7f3caaf0f700 1 -- 172.21.15.178:0/1071067718 wait complete. 2024-03-28T02:02:47.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:47 smithi153 ceph-mon[36228]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:47.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:47.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:47.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T02:02:47.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T02:02:47.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:48.585 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:48 smithi178 ceph-mon[36521]: Reconfiguring mon.c (monmap changed)... 2024-03-28T02:02:48.585 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:48 smithi178 ceph-mon[36521]: Reconfiguring daemon mon.c on smithi178 2024-03-28T02:02:48.585 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:48 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:48.613 DEBUG:teuthology.orchestra.run.smithi153:mgr.y> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T02:02:48.616 DEBUG:teuthology.orchestra.run.smithi178:mgr.z> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T02:02:48.618 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T02:02:48.618 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:02:48.619 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T02:02:48.638 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T02:02:48.638 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-03-28T02:02:48.696 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T02:02:48.696 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:48.696 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-28T02:02:48.696 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:48.696 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:48.696 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-28 02:01:54.654006636 +0000 2024-03-28T02:02:48.696 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-28 01:56:43.499324748 +0000 2024-03-28T02:02:48.696 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-28 01:56:43.499324748 +0000 2024-03-28T02:02:48.696 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-28 01:56:43.499324748 +0000 2024-03-28T02:02:48.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T02:02:48.763 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:48 smithi106 ceph-mon[31965]: Reconfiguring mon.c (monmap changed)... 2024-03-28T02:02:48.763 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:48 smithi106 ceph-mon[31965]: Reconfiguring daemon mon.c on smithi178 2024-03-28T02:02:48.763 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:48 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:48.766 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-28T02:02:48.766 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-28T02:02:48.766 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000293921 s, 1.7 MB/s 2024-03-28T02:02:48.768 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T02:02:48.827 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-03-28T02:02:48.884 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T02:02:48.884 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:48.885 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-28T02:02:48.885 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:48.885 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:48.885 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-28 02:01:54.685005865 +0000 2024-03-28T02:02:48.885 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-28 01:56:43.865315500 +0000 2024-03-28T02:02:48.885 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-28 01:56:43.865315500 +0000 2024-03-28T02:02:48.885 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-28 01:56:43.865315500 +0000 2024-03-28T02:02:48.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T02:02:48.955 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-28T02:02:48.955 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-28T02:02:48.956 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000250807 s, 2.0 MB/s 2024-03-28T02:02:48.957 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T02:02:48.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:48 smithi153 ceph-mon[36228]: Reconfiguring mon.c (monmap changed)... 2024-03-28T02:02:48.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:48 smithi153 ceph-mon[36228]: Reconfiguring daemon mon.c on smithi178 2024-03-28T02:02:48.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:48 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:49.015 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-03-28T02:02:49.073 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T02:02:49.074 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:49.074 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-28T02:02:49.074 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:49.074 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:49.074 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-28 02:01:54.716005095 +0000 2024-03-28T02:02:49.074 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-28 01:56:44.209306808 +0000 2024-03-28T02:02:49.074 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-28 01:56:44.209306808 +0000 2024-03-28T02:02:49.074 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-28 01:56:44.209306808 +0000 2024-03-28T02:02:49.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T02:02:49.144 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-28T02:02:49.144 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-28T02:02:49.144 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000210974 s, 2.4 MB/s 2024-03-28T02:02:49.145 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T02:02:49.203 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-03-28T02:02:49.259 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T02:02:49.259 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:49.259 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-28T02:02:49.259 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:49.260 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:49.260 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-28 02:01:54.752004200 +0000 2024-03-28T02:02:49.260 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-28 01:56:44.528298747 +0000 2024-03-28T02:02:49.260 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-28 01:56:44.528298747 +0000 2024-03-28T02:02:49.260 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-28 01:56:44.528298747 +0000 2024-03-28T02:02:49.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T02:02:49.327 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-28T02:02:49.328 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-28T02:02:49.328 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000205727 s, 2.5 MB/s 2024-03-28T02:02:49.329 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T02:02:49.386 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-28T02:02:49.386 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T02:02:49.404 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T02:02:49.405 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-28T02:02:49.462 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T02:02:49.463 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:49.463 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-28T02:02:49.463 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:49.463 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:49.463 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-28 02:02:35.781131101 +0000 2024-03-28T02:02:49.463 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-28 01:55:00.347786791 +0000 2024-03-28T02:02:49.463 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-28 01:55:00.347786791 +0000 2024-03-28T02:02:49.463 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-28 01:55:00.347786791 +0000 2024-03-28T02:02:49.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T02:02:49.532 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-28T02:02:49.532 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-28T02:02:49.532 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000227205 s, 2.3 MB/s 2024-03-28T02:02:49.533 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T02:02:49.591 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-28T02:02:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T02:02:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:49.647 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-28T02:02:49.647 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:49.647 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:49.647 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-28 02:02:35.782131083 +0000 2024-03-28T02:02:49.647 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-28 01:55:00.695780229 +0000 2024-03-28T02:02:49.647 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-28 01:55:00.695780229 +0000 2024-03-28T02:02:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-28 01:55:00.695780229 +0000 2024-03-28T02:02:49.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T02:02:49.714 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-28T02:02:49.715 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-28T02:02:49.715 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000192672 s, 2.7 MB/s 2024-03-28T02:02:49.716 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T02:02:49.773 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-28T02:02:49.828 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T02:02:49.828 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:49.828 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T02:02:49.828 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:49.828 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:49.828 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-28 02:02:35.782131083 +0000 2024-03-28T02:02:49.828 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-28 01:55:01.026773988 +0000 2024-03-28T02:02:49.828 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-28 01:55:01.026773988 +0000 2024-03-28T02:02:49.828 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-28 01:55:01.026773988 +0000 2024-03-28T02:02:49.829 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T02:02:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:49 smithi106 ceph-mon[31965]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi106=x;smithi153=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:49 smithi106 ceph-mon[31965]: Saving service mgr spec with placement smithi106=x;smithi153=y;smithi178=z;count:3 2024-03-28T02:02:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:49 smithi106 ceph-mon[31965]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:49.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:49 smithi178 ceph-mon[36521]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi106=x;smithi153=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:49 smithi178 ceph-mon[36521]: Saving service mgr spec with placement smithi106=x;smithi153=y;smithi178=z;count:3 2024-03-28T02:02:49.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:49 smithi178 ceph-mon[36521]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:49.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:49.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:49.892 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:49 smithi153 ceph-mon[36228]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi106=x;smithi153=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:49.892 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:49 smithi153 ceph-mon[36228]: Saving service mgr spec with placement smithi106=x;smithi153=y;smithi178=z;count:3 2024-03-28T02:02:49.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:49 smithi153 ceph-mon[36228]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:49.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:49.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:49.895 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-28T02:02:49.895 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-28T02:02:49.895 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000170408 s, 3.0 MB/s 2024-03-28T02:02:49.896 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T02:02:49.951 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-28T02:02:50.007 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T02:02:50.007 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:50.007 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-28T02:02:50.007 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:50.007 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:50.007 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-28 02:02:35.783131064 +0000 2024-03-28T02:02:50.007 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-28 01:55:01.363767633 +0000 2024-03-28T02:02:50.007 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-28 01:55:01.363767633 +0000 2024-03-28T02:02:50.007 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-28 01:55:01.363767633 +0000 2024-03-28T02:02:50.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T02:02:50.072 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-28T02:02:50.072 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-28T02:02:50.072 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000201155 s, 2.5 MB/s 2024-03-28T02:02:50.074 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T02:02:50.129 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-28T02:02:50.129 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T02:02:50.146 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T02:02:50.146 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-03-28T02:02:50.202 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T02:02:50.202 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:50.202 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-28T02:02:50.202 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:50.202 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:50.202 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-28 02:02:26.297309194 +0000 2024-03-28T02:02:50.202 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-28 01:55:00.341329654 +0000 2024-03-28T02:02:50.202 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-28 01:55:00.341329654 +0000 2024-03-28T02:02:50.202 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-28 01:55:00.341329654 +0000 2024-03-28T02:02:50.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T02:02:50.269 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-28T02:02:50.269 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-28T02:02:50.269 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 9.7055e-05 s, 5.3 MB/s 2024-03-28T02:02:50.271 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T02:02:50.327 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-03-28T02:02:50.381 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T02:02:50.381 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:50.381 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-28T02:02:50.381 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:50.381 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:50.382 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-28 02:02:26.298309176 +0000 2024-03-28T02:02:50.382 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-28 01:55:00.673323456 +0000 2024-03-28T02:02:50.382 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-28 01:55:00.673323456 +0000 2024-03-28T02:02:50.382 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-28 01:55:00.673323456 +0000 2024-03-28T02:02:50.382 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T02:02:50.447 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-28T02:02:50.447 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-28T02:02:50.447 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 9.4632e-05 s, 5.4 MB/s 2024-03-28T02:02:50.448 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T02:02:50.504 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-03-28T02:02:50.560 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T02:02:50.560 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:50.560 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T02:02:50.560 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:50.560 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:50.560 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-28 02:02:26.298309176 +0000 2024-03-28T02:02:50.560 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-28 01:55:00.980317724 +0000 2024-03-28T02:02:50.560 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-28 01:55:00.980317724 +0000 2024-03-28T02:02:50.560 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-28 01:55:00.980317724 +0000 2024-03-28T02:02:50.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T02:02:50.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:50.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:50.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:50.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:50.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T02:02:50.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T02:02:50.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T02:02:50.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:50.627 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-28T02:02:50.627 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-28T02:02:50.627 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000109978 s, 4.7 MB/s 2024-03-28T02:02:50.628 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T02:02:50.684 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-03-28T02:02:50.739 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T02:02:50.739 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T02:02:50.739 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-28T02:02:50.739 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:02:50.739 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T02:02:50.739 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-28 02:02:26.298309176 +0000 2024-03-28T02:02:50.739 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-28 01:55:01.294311861 +0000 2024-03-28T02:02:50.739 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-28 01:55:01.294311861 +0000 2024-03-28T02:02:50.739 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-28 01:55:01.294311861 +0000 2024-03-28T02:02:50.740 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T02:02:50.803 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-28T02:02:50.803 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-28T02:02:50.803 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 8.7405e-05 s, 5.9 MB/s 2024-03-28T02:02:50.805 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T02:02:50.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:50.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:50.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:50.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:50.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T02:02:50.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T02:02:50.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T02:02:50.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:50.859 INFO:tasks.cephadm:Deploying osd.0 on smithi106 with /dev/vg_nvme/lv_4... 2024-03-28T02:02:50.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T02:02:50.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:50.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:50.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:50.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:50.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T02:02:50.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T02:02:50.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T02:02:50.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:51.091 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:02:51.745 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:51 smithi178 ceph-mon[36521]: Deploying daemon mgr.z on smithi178 2024-03-28T02:02:51.745 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:51 smithi178 ceph-mon[36521]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:51.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:51 smithi106 ceph-mon[31965]: Deploying daemon mgr.z on smithi178 2024-03-28T02:02:51.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:51 smithi106 ceph-mon[31965]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:51.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:51 smithi153 ceph-mon[36228]: Deploying daemon mgr.z on smithi178 2024-03-28T02:02:51.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:51 smithi153 ceph-mon[36228]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:52.097 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:52 smithi178 systemd[1]: Starting Ceph mgr.z for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:02:52.373 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:52 smithi178 podman[38369]: 2024-03-28 02:02:52.220287423 +0000 UTC m=+0.017490906 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:02:52.373 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:52 smithi178 podman[38369]: 2024-03-28 02:02:52.322300771 +0000 UTC m=+0.119504257 container create 334ab3dddb836c71c2fb9ee4bdf49338a3c7bac7e8fdcbb613de51cd988e9490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T02:02:52.873 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:52 smithi178 podman[38369]: 2024-03-28 02:02:52.558325106 +0000 UTC m=+0.355528589 container init 334ab3dddb836c71c2fb9ee4bdf49338a3c7bac7e8fdcbb613de51cd988e9490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-28T02:02:52.873 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:52 smithi178 podman[38369]: 2024-03-28 02:02:52.560873325 +0000 UTC m=+0.358076802 container start 334ab3dddb836c71c2fb9ee4bdf49338a3c7bac7e8fdcbb613de51cd988e9490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T02:02:52.873 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:52 smithi178 ceph-mgr[38388]: -- 172.21.15.178:0/2448858865 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592d2e5e5a0 con 0x5592d2e33000 2024-03-28T02:02:52.873 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:52 smithi178 bash[38369]: 334ab3dddb836c71c2fb9ee4bdf49338a3c7bac7e8fdcbb613de51cd988e9490 2024-03-28T02:02:52.873 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:52 smithi178 systemd[1]: Started Ceph mgr.z for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:02:52.950 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T02:02:52.951 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T02:02:52.951 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-28T02:02:52.951 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-28T02:02:52.951 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131391 s, 798 MB/s 2024-03-28T02:02:52.951 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-28T02:02:53.177 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:53.092+0000 7fbde3a3f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T02:02:53.177 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:53.182+0000 7fbde3a3f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T02:02:53.506 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:53 smithi106 ceph-mon[31965]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:53.506 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:53.506 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:53.506 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:53.506 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T02:02:53.507 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T02:02:53.507 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T02:02:53.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_4 2024-03-28T02:02:53.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:53 smithi153 ceph-mon[36228]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:53.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:53.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:53.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:53.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T02:02:53.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T02:02:53.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T02:02:53.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:53.803 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:02:53.804 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-mon[36521]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:53.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:53.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:53.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:53.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T02:02:53.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T02:02:53.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T02:02:53.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:53.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:54.122 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:53.809+0000 7fbde3a3f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T02:02:54.123 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:53 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:53.899+0000 7fbde3a3f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T02:02:54.123 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:54.037+0000 7fbde3a3f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T02:02:54.461 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:54.129+0000 7fbde3a3f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T02:02:54.462 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:54.209+0000 7fbde3a3f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T02:02:54.462 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:54.466+0000 7fbde3a3f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T02:02:54.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:54 smithi106 ceph-mon[31965]: Deploying daemon mgr.y on smithi153 2024-03-28T02:02:54.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:54 smithi153 ceph-mon[36228]: Deploying daemon mgr.y on smithi153 2024-03-28T02:02:54.744 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:54 smithi178 ceph-mon[36521]: Deploying daemon mgr.y on smithi153 2024-03-28T02:02:54.744 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:54.575+0000 7fbde3a3f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T02:02:54.745 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:54.658+0000 7fbde3a3f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T02:02:54.745 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:54.749+0000 7fbde3a3f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T02:02:55.122 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:54.924+0000 7fbde3a3f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T02:02:55.177 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:54 smithi153 systemd[1]: Starting Ceph mgr.y for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:02:55.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.284+0000 7f5f25137700 1 -- 172.21.15.106:0/2487695557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f20076140 msgr2=0x7f5f200745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:55.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.284+0000 7f5f25137700 1 --2- 172.21.15.106:0/2487695557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f20076140 0x7f5f200745a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009990 tx=0x7f5f10009ca0 comp rx=0 tx=0).stop 2024-03-28T02:02:55.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.284+0000 7f5f25137700 1 -- 172.21.15.106:0/2487695557 shutdown_connections 2024-03-28T02:02:55.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.284+0000 7f5f25137700 1 --2- 172.21.15.106:0/2487695557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f2010a0c0 0x7f5f2010c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:55.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.284+0000 7f5f25137700 1 --2- 172.21.15.106:0/2487695557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f20076140 0x7f5f200745a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:55.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.284+0000 7f5f25137700 1 --2- 172.21.15.106:0/2487695557 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f20074b70 0x7f5f20074ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:55.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.284+0000 7f5f25137700 1 -- 172.21.15.106:0/2487695557 >> 172.21.15.106:0/2487695557 conn(0x7f5f200fd7a0 msgr2=0x7f5f200ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:02:55.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.284+0000 7f5f25137700 1 -- 172.21.15.106:0/2487695557 shutdown_connections 2024-03-28T02:02:55.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.284+0000 7f5f25137700 1 -- 172.21.15.106:0/2487695557 wait complete. 2024-03-28T02:02:55.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f25137700 1 Processor -- start 2024-03-28T02:02:55.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f25137700 1 -- start start 2024-03-28T02:02:55.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f25137700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f20074b70 0x7f5f2019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:55.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f25137700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f20076140 0x7f5f2019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:55.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f25137700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f2010a0c0 0x7f5f201a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f25137700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2010ff90 con 0x7f5f2010a0c0 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f25137700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2010fe10 con 0x7f5f20076140 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f25137700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f20110110 con 0x7f5f20074b70 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f1f59e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f2010a0c0 0x7f5f201a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f1659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f20076140 0x7f5f2019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f1ed9d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f20074b70 0x7f5f2019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f1f59e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f2010a0c0 0x7f5f201a1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49282/0 (socket says 172.21.15.106:49282) 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f1659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f20076140 0x7f5f2019aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:38332/0 (socket says 172.21.15.106:38332) 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.285+0000 7f5f1f59e700 1 -- 172.21.15.106:0/350640385 learned_addr learned my addr 172.21.15.106:0/350640385 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1f59e700 1 -- 172.21.15.106:0/350640385 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f20074b70 msgr2=0x7f5f2019a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1f59e700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f20074b70 0x7f5f2019a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1f59e700 1 -- 172.21.15.106:0/350640385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f20076140 msgr2=0x7f5f2019aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:02:55.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1f59e700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f20076140 0x7f5f2019aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1f59e700 1 -- 172.21.15.106:0/350640385 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f201a2210 con 0x7f5f2010a0c0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1ed9d700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f20074b70 0x7f5f2019a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1f59e700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f2010a0c0 0x7f5f201a1b70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5f18007f10 tx=0x7f5f1800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1cd99700 1 -- 172.21.15.106:0/350640385 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18004bb0 con 0x7f5f2010a0c0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1cd99700 1 -- 172.21.15.106:0/350640385 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f18004d50 con 0x7f5f2010a0c0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f201a2500 con 0x7f5f2010a0c0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f1cd99700 1 -- 172.21.15.106:0/350640385 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f180059f0 con 0x7f5f2010a0c0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.286+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f201a2a90 con 0x7f5f2010a0c0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.287+0000 7f5f1cd99700 1 -- 172.21.15.106:0/350640385 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f5f1801a020 con 0x7f5f2010a0c0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.287+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f20068e50 con 0x7f5f2010a0c0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.287+0000 7f5f1cd99700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5f0c039200 0x7f5f0c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.287+0000 7f5f1cd99700 1 -- 172.21.15.106:0/350640385 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5f1804b8d0 con 0x7f5f2010a0c0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.287+0000 7f5f1ed9d700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5f0c039200 0x7f5f0c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:02:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.288+0000 7f5f1ed9d700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5f0c039200 0x7f5f0c03b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000b2e0 tx=0x7f5f10005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:02:55.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.292+0000 7f5f1cd99700 1 -- 172.21.15.106:0/350640385 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f18010070 con 0x7f5f2010a0c0 2024-03-28T02:02:55.431 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:55 smithi153 podman[37591]: 2024-03-28 02:02:55.086608147 +0000 UTC m=+0.018052786 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:02:55.431 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:55 smithi153 podman[37591]: 2024-03-28 02:02:55.184859187 +0000 UTC m=+0.116303804 container create 9442b8a6cda58dd51d258aaaaecc0611502e41a732cadb555b1b6052889f0dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T02:02:55.431 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:55 smithi153 podman[37591]: 2024-03-28 02:02:55.362521663 +0000 UTC m=+0.293966282 container init 9442b8a6cda58dd51d258aaaaecc0611502e41a732cadb555b1b6052889f0dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-28T02:02:55.431 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:55 smithi153 podman[37591]: 2024-03-28 02:02:55.365216947 +0000 UTC m=+0.296661563 container start 9442b8a6cda58dd51d258aaaaecc0611502e41a732cadb555b1b6052889f0dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:02:55.431 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:55 smithi153 ceph-mgr[37610]: -- 172.21.15.153:0/333422941 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564902f565a0 con 0x564902f2ac00 2024-03-28T02:02:55.431 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:55 smithi153 bash[37591]: 9442b8a6cda58dd51d258aaaaecc0611502e41a732cadb555b1b6052889f0dd6 2024-03-28T02:02:55.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:02:55.505+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f200625d0 con 0x7f5f0c039200 2024-03-28T02:02:55.733 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:55 smithi153 systemd[1]: Started Ceph mgr.y for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:02:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:55 smithi153 ceph-mon[36228]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:55.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:55 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:55.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:55 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:02:55.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:55 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:55.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:55 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:02:55.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:55 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:55.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:55 smithi106 ceph-mon[31965]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:55.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:55.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:02:55.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:55.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:02:55.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:55 smithi178 ceph-mon[36521]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:55.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:55 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:55.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:55 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:02:55.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:55 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:55.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:55 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:02:55.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:55 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:56.233 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:55 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:55.931+0000 7fd7e5e34200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T02:02:56.233 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:56 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:56.028+0000 7fd7e5e34200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T02:02:56.372 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:55 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:55.897+0000 7fbde3a3f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T02:02:56.373 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:56.031+0000 7fbde3a3f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T02:02:56.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:56 smithi178 ceph-mon[36521]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:56.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:56.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:56.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:56.678 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:56.393+0000 7fbde3a3f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T02:02:56.678 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:56.485+0000 7fbde3a3f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T02:02:56.679 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:56.684+0000 7fbde3a3f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:02:56.702 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:56 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:56.708+0000 7fd7e5e34200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T02:02:56.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:56 smithi153 ceph-mon[36228]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:56.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:56.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:56.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:56.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:56 smithi106 ceph-mon[31965]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:02:56.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:56.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:56.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:02:56.983 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:56 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:56.800+0000 7fd7e5e34200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T02:02:56.983 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:56 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:56.951+0000 7fd7e5e34200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T02:02:57.122 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:56.870+0000 7fbde3a3f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T02:02:57.389 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:57.048+0000 7fd7e5e34200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T02:02:57.389 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:57.132+0000 7fd7e5e34200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T02:02:57.389 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:57.394+0000 7fd7e5e34200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T02:02:57.404 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:57.153+0000 7fbde3a3f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:02:57.404 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:57.410+0000 7fbde3a3f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T02:02:57.678 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:57.504+0000 7fd7e5e34200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T02:02:57.678 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:57.591+0000 7fd7e5e34200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T02:02:57.678 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:57.684+0000 7fd7e5e34200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:57.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:57.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.872 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:57.639+0000 7fbde3a3f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:02:57.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:02:57.983 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:57.859+0000 7fd7e5e34200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T02:02:58.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:58 smithi153 ceph-mon[36228]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T02:02:58.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:58 smithi153 ceph-mon[36228]: Reconfiguring daemon mgr.x on smithi106 2024-03-28T02:02:58.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:58 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3654821981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57987f85-e72f-4e07-b777-cc2243ffbbe7"}]: dispatch 2024-03-28T02:02:58.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:58 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3654821981' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57987f85-e72f-4e07-b777-cc2243ffbbe7"}]': finished 2024-03-28T02:02:58.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:58 smithi153 ceph-mon[36228]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T02:02:58.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:58 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:02:58.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:58 smithi106 ceph-mon[31965]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T02:02:58.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:58 smithi106 ceph-mon[31965]: Reconfiguring daemon mgr.x on smithi106 2024-03-28T02:02:58.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:58 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3654821981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57987f85-e72f-4e07-b777-cc2243ffbbe7"}]: dispatch 2024-03-28T02:02:58.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:58 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3654821981' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57987f85-e72f-4e07-b777-cc2243ffbbe7"}]': finished 2024-03-28T02:02:58.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:58 smithi106 ceph-mon[31965]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T02:02:58.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:02:58.872 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:58 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:58.413+0000 7fbde3a3f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T02:02:58.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:58 smithi178 ceph-mon[36521]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T02:02:58.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:58 smithi178 ceph-mon[36521]: Reconfiguring daemon mgr.x on smithi106 2024-03-28T02:02:58.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:58 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3654821981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57987f85-e72f-4e07-b777-cc2243ffbbe7"}]: dispatch 2024-03-28T02:02:58.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:58 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3654821981' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57987f85-e72f-4e07-b777-cc2243ffbbe7"}]': finished 2024-03-28T02:02:58.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:58 smithi178 ceph-mon[36521]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T02:02:58.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:58 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:02:59.233 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:58 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:58.830+0000 7fd7e5e34200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T02:02:59.233 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:58 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:58.960+0000 7fd7e5e34200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T02:02:59.268 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:58 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:58.974+0000 7fbde3a3f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T02:02:59.268 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:59 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:59.067+0000 7fbde3a3f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T02:02:59.268 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:02:59 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:02:59.274+0000 7fbde3a3f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T02:02:59.576 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:59.297+0000 7fd7e5e34200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T02:02:59.576 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:59.386+0000 7fd7e5e34200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T02:02:59.577 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:59.582+0000 7fd7e5e34200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:02:59.577 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-mon[36228]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:59.577 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2663396362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:02:59.577 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-mon[36228]: Standby manager daemon z started 2024-03-28T02:02:59.577 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T02:02:59.577 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:02:59.577 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T02:02:59.577 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:02:59.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:59 smithi178 ceph-mon[36521]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:59.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:59 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2663396362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:02:59.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:59 smithi178 ceph-mon[36521]: Standby manager daemon z started 2024-03-28T02:02:59.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:59 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T02:02:59.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:59 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:02:59.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:59 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T02:02:59.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:02:59 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:02:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:59 smithi106 ceph-mon[31965]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:02:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:59 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2663396362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:02:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:59 smithi106 ceph-mon[31965]: Standby manager daemon z started 2024-03-28T02:02:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:59 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T02:02:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:59 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:02:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:59 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T02:02:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:02:59 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.178:0/3412307706' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:02:59.983 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:02:59 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:02:59.766+0000 7fd7e5e34200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T02:03:00.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:00.111+0000 7f5f1cd99700 1 -- 172.21.15.106:0/350640385 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7f5f1804dcc0 con 0x7f5f2010a0c0 2024-03-28T02:03:00.303 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:03:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:03:00.048+0000 7fd7e5e34200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:03:00.303 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:03:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:03:00.309+0000 7fd7e5e34200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T02:03:00.733 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:03:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:03:00.544+0000 7fd7e5e34200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T02:03:01.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:01 smithi178 ceph-mon[36521]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-28T02:03:01.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:01 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T02:03:01.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:01 smithi178 ceph-mon[36521]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:01 smithi153 ceph-mon[36228]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-28T02:03:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:01 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T02:03:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:01 smithi153 ceph-mon[36228]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:01.484 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:03:01 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:03:01.343+0000 7fd7e5e34200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T02:03:01.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:01 smithi106 ceph-mon[31965]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-28T02:03:01.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:01 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T02:03:01.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:01 smithi106 ceph-mon[31965]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:02.217 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:03:01 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:03:01.913+0000 7fd7e5e34200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T02:03:02.217 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:03:02 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:03:02.009+0000 7fd7e5e34200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T02:03:02.217 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:03:02 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:03:02.223+0000 7fd7e5e34200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T02:03:02.289 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:02.295+0000 7f5f1cd99700 1 -- 172.21.15.106:0/350640385 <== mon.0 v2:172.21.15.106:3300/0 8 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f5f1803ad70 con 0x7f5f2010a0c0 2024-03-28T02:03:02.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:02 smithi153 ceph-mon[36228]: Standby manager daemon y started 2024-03-28T02:03:02.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:02 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T02:03:02.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:02 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:03:02.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:02 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T02:03:02.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:02 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:03:02.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:02 smithi106 ceph-mon[31965]: Standby manager daemon y started 2024-03-28T02:03:02.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:02 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T02:03:02.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:02 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:03:02.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:02 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T02:03:02.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:02 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:03:02.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:02 smithi178 ceph-mon[36521]: Standby manager daemon y started 2024-03-28T02:03:02.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:02 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T02:03:02.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:02 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:03:02.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:02 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T02:03:02.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:02 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.153:0/2063887823' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:03:03.546 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:03 smithi106 ceph-mon[31965]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:03.547 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:03 smithi106 ceph-mon[31965]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-28T02:03:03.547 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:03 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T02:03:03.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:03 smithi178 ceph-mon[36521]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:03.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:03 smithi178 ceph-mon[36521]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-28T02:03:03.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:03 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T02:03:03.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:03 smithi153 ceph-mon[36228]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:03.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:03 smithi153 ceph-mon[36228]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-28T02:03:03.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:03 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T02:03:04.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:04.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:04.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:04.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:04.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:04.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:04.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:04 smithi106 ceph-mon[31965]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:04.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:04 smithi178 ceph-mon[36521]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:04.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:04.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:04.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:04.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:04.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:04.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:04.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:04 smithi153 ceph-mon[36228]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:06.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T02:03:06.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:06.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T02:03:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:06.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T02:03:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:07.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:07 smithi106 ceph-mon[31965]: Deploying daemon osd.0 on smithi106 2024-03-28T02:03:07.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:07 smithi106 ceph-mon[31965]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:07 smithi153 ceph-mon[36228]: Deploying daemon osd.0 on smithi106 2024-03-28T02:03:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:07 smithi153 ceph-mon[36228]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:07.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:07 smithi178 ceph-mon[36521]: Deploying daemon osd.0 on smithi106 2024-03-28T02:03:07.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:07 smithi178 ceph-mon[36521]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:09.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:09 smithi106 ceph-mon[31965]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:09.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:09 smithi178 ceph-mon[36521]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:09.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:09 smithi153 ceph-mon[36228]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:11.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:11 smithi106 ceph-mon[31965]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:11.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:11 smithi178 ceph-mon[36521]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:11 smithi153 ceph-mon[36228]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:12.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:12.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:12.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:12.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:12.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:13.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:13 smithi106 ceph-mon[31965]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:13.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:13 smithi153 ceph-mon[36228]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:13.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:13 smithi178 ceph-mon[36521]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:14.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.728+0000 7f5f1cd99700 1 -- 172.21.15.106:0/350640385 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5f200625d0 con 0x7f5f0c039200 2024-03-28T02:03:14.721 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 0 on host 'smithi106' 2024-03-28T02:03:14.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5f0c039200 msgr2=0x7f5f0c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5f0c039200 0x7f5f0c03b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000b2e0 tx=0x7f5f10005fd0 comp rx=0 tx=0).stop 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f2010a0c0 msgr2=0x7f5f201a1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f2010a0c0 0x7f5f201a1b70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5f18007f10 tx=0x7f5f1800d3b0 comp rx=0 tx=0).stop 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 shutdown_connections 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5f0c039200 0x7f5f0c03b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f2010a0c0 0x7f5f201a1b70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f20074b70 0x7f5f2019a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 --2- 172.21.15.106:0/350640385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f20076140 0x7f5f2019aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 >> 172.21.15.106:0/350640385 conn(0x7f5f200fd7a0 msgr2=0x7f5f20106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 shutdown_connections 2024-03-28T02:03:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:03:14.730+0000 7f5f25137700 1 -- 172.21.15.106:0/350640385 wait complete. 2024-03-28T02:03:15.401 DEBUG:teuthology.orchestra.run.smithi106:osd.0> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T02:03:15.403 INFO:tasks.cephadm:Deploying osd.1 on smithi153 with /dev/vg_nvme/lv_4... 2024-03-28T02:03:15.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T02:03:15.646 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:03:15.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:15 smithi178 ceph-mon[36521]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:15.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:15 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:15 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:15 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:15 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:15 smithi178 ceph-mon[36521]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T02:03:15.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:15 smithi153 ceph-mon[36228]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:15.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:15 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:15 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:15 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:15 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:15 smithi153 ceph-mon[36228]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T02:03:15.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:15 smithi106 ceph-mon[31965]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:15.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:15 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:15 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:15 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:15 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:15.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:15 smithi106 ceph-mon[31965]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T02:03:17.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:17 smithi153 ceph-mon[36228]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T02:03:17.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:17 smithi153 ceph-mon[36228]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T02:03:17.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:17 smithi153 ceph-mon[36228]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:03:17.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:17.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:17 smithi153 ceph-mon[36228]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:17.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:17 smithi106 ceph-mon[31965]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T02:03:17.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:17 smithi106 ceph-mon[31965]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T02:03:17.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:17 smithi106 ceph-mon[31965]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:03:17.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:17.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:17 smithi106 ceph-mon[31965]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:17.334 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:03:17 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0[46711]: 2024-03-28T02:03:17.010+0000 7fccf6250700 -1 osd.0 0 waiting for initial osdmap 2024-03-28T02:03:17.334 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:03:17 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0[46711]: 2024-03-28T02:03:17.017+0000 7fccf2043700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:03:17.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:17 smithi178 ceph-mon[36521]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T02:03:17.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:17 smithi178 ceph-mon[36521]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T02:03:17.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:17 smithi178 ceph-mon[36521]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:03:17.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:17.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:17 smithi178 ceph-mon[36521]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:17.536 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T02:03:17.536 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T02:03:17.536 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-28T02:03:17.536 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-28T02:03:17.536 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120308 s, 872 MB/s 2024-03-28T02:03:17.537 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-28T02:03:18.120 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:18 smithi153 ceph-mon[36228]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:03:18.121 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:18 smithi153 ceph-mon[36228]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T02:03:18.121 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:18 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:18.121 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:18 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:18.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-03-28T02:03:18.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:18 smithi106 ceph-mon[31965]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:03:18.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:18 smithi106 ceph-mon[31965]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T02:03:18.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:18 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:18.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:18 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:18.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:18 smithi178 ceph-mon[36521]: from='osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:03:18.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:18 smithi178 ceph-mon[36521]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T02:03:18.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:18 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:18.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:18 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:18.394 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:03:19.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:03:19.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:03:19.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:19.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] boot 2024-03-28T02:03:19.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T02:03:19.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:19.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:19.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi106 2024-03-28T02:03:19.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:03:19.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi106 to 17088M 2024-03-28T02:03:19.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:19.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:19.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:19.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] boot 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi106 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi106 to 17088M 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:19.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:19.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:03:19.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:03:19.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:19.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: osd.0 [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] boot 2024-03-28T02:03:19.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T02:03:19.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:03:19.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T02:03:19.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi106 2024-03-28T02:03:19.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:03:19.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi106 to 17088M 2024-03-28T02:03:19.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:19.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:19.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:19.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:19.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.871+0000 7f0a5a30c700 1 -- 172.21.15.153:0/52074915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a54102c60 msgr2=0x7f0a54109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:19.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.871+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/52074915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a54102c60 0x7f0a54109510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00b0d0 tx=0x7f0a4c00b3e0 comp rx=0 tx=0).stop 2024-03-28T02:03:19.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.871+0000 7f0a5a30c700 1 -- 172.21.15.153:0/52074915 shutdown_connections 2024-03-28T02:03:19.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.871+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/52074915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a54102c60 0x7f0a54109510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.871+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/52074915 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a541010c0 0x7f0a541014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.871+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/52074915 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a541022c0 0x7f0a54102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.871+0000 7f0a5a30c700 1 -- 172.21.15.153:0/52074915 >> 172.21.15.153:0/52074915 conn(0x7f0a540fc660 msgr2=0x7f0a540feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.871+0000 7f0a5a30c700 1 -- 172.21.15.153:0/52074915 shutdown_connections 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.871+0000 7f0a5a30c700 1 -- 172.21.15.153:0/52074915 wait complete. 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a5a30c700 1 Processor -- start 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a5a30c700 1 -- start start 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a5a30c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a541010c0 0x7f0a5419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a5a30c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a541022c0 0x7f0a5419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a5a30c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a54102c60 0x7f0a541a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a5a30c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5410cd00 con 0x7f0a54102c60 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a5a30c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5410cb80 con 0x7f0a541010c0 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a5a30c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5410ce80 con 0x7f0a541022c0 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a53fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a541010c0 0x7f0a5419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:03:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a53fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a541010c0 0x7f0a5419a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52066/0 (socket says 172.21.15.153:52066) 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a53fff700 1 -- 172.21.15.153:0/748139198 learned_addr learned my addr 172.21.15.153:0/748139198 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a588a9700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a54102c60 0x7f0a541a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.872+0000 7f0a537fe700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a541022c0 0x7f0a5419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a53fff700 1 -- 172.21.15.153:0/748139198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a541022c0 msgr2=0x7f0a5419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a53fff700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a541022c0 0x7f0a5419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a53fff700 1 -- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a54102c60 msgr2=0x7f0a541a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a53fff700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a54102c60 0x7f0a541a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a53fff700 1 -- 172.21.15.153:0/748139198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a541a2250 con 0x7f0a541010c0 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a53fff700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a541010c0 0x7f0a5419a5f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00d980 tx=0x7f0a3c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a517fa700 1 -- 172.21.15.153:0/748139198 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c017070 con 0x7f0a541010c0 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a517fa700 1 -- 172.21.15.153:0/748139198 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a3c009e10 con 0x7f0a541010c0 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a541a2540 con 0x7f0a541010c0 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a517fa700 1 -- 172.21.15.153:0/748139198 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c00f620 con 0x7f0a541010c0 2024-03-28T02:03:19.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.873+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a541a2a30 con 0x7f0a541010c0 2024-03-28T02:03:19.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.874+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a54068e50 con 0x7f0a541010c0 2024-03-28T02:03:19.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.875+0000 7f0a517fa700 1 -- 172.21.15.153:0/748139198 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f0a3c010710 con 0x7f0a541010c0 2024-03-28T02:03:19.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.876+0000 7f0a517fa700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0a400a2800 0x7f0a400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:03:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.876+0000 7f0a517fa700 1 -- 172.21.15.153:0/748139198 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f0a3c0ccb50 con 0x7f0a541010c0 2024-03-28T02:03:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.876+0000 7f0a537fe700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0a400a2800 0x7f0a400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:03:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.876+0000 7f0a537fe700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0a400a2800 0x7f0a400a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0a44005950 tx=0x7f0a440058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:03:19.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:19.880+0000 7f0a517fa700 1 -- 172.21.15.153:0/748139198 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a3c09b660 con 0x7f0a541010c0 2024-03-28T02:03:20.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:20.087+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a540fe820 con 0x7f0a400a2800 2024-03-28T02:03:20.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:20 smithi106 ceph-mon[31965]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T02:03:20.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:20.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:20.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:20.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:20 smithi178 ceph-mon[36521]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T02:03:20.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:20.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:20.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:20.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:20 smithi153 ceph-mon[36228]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T02:03:20.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:20.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:20.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:21.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:21 smithi153 ceph-mon[36228]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:03:21.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:03:21.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:03:21.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:21.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:21 smithi153 ceph-mon[36228]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:21.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:21 smithi106 ceph-mon[31965]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:03:21.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:03:21.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:03:21.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:21.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:21 smithi106 ceph-mon[31965]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:21.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:21 smithi178 ceph-mon[36521]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:03:21.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:03:21.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:03:21.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:21.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:21 smithi178 ceph-mon[36521]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:23.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:23 smithi106 ceph-mon[31965]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:23 smithi106 ceph-mon[31965]: from='client.? 172.21.15.153:0/3736498698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45e0b5df-6da6-4f24-aaf4-2d1531425742"}]: dispatch 2024-03-28T02:03:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:23 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45e0b5df-6da6-4f24-aaf4-2d1531425742"}]: dispatch 2024-03-28T02:03:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:23 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45e0b5df-6da6-4f24-aaf4-2d1531425742"}]': finished 2024-03-28T02:03:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:23 smithi106 ceph-mon[31965]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T02:03:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:23 smithi106 ceph-mon[31965]: from='client.? 172.21.15.153:0/639517621' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:03:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:23 smithi178 ceph-mon[36521]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:23.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:23 smithi178 ceph-mon[36521]: from='client.? 172.21.15.153:0/3736498698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45e0b5df-6da6-4f24-aaf4-2d1531425742"}]: dispatch 2024-03-28T02:03:23.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:23 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45e0b5df-6da6-4f24-aaf4-2d1531425742"}]: dispatch 2024-03-28T02:03:23.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:23 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45e0b5df-6da6-4f24-aaf4-2d1531425742"}]': finished 2024-03-28T02:03:23.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:23 smithi178 ceph-mon[36521]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T02:03:23.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:23.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:23 smithi178 ceph-mon[36521]: from='client.? 172.21.15.153:0/639517621' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:03:23.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:23 smithi153 ceph-mon[36228]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:23.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:23 smithi153 ceph-mon[36228]: from='client.? 172.21.15.153:0/3736498698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45e0b5df-6da6-4f24-aaf4-2d1531425742"}]: dispatch 2024-03-28T02:03:23.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:23 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45e0b5df-6da6-4f24-aaf4-2d1531425742"}]: dispatch 2024-03-28T02:03:23.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:23 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45e0b5df-6da6-4f24-aaf4-2d1531425742"}]': finished 2024-03-28T02:03:23.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:23 smithi153 ceph-mon[36228]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T02:03:23.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:23.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:23 smithi153 ceph-mon[36228]: from='client.? 172.21.15.153:0/639517621' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:03:25.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:25 smithi106 ceph-mon[31965]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:25.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:25 smithi178 ceph-mon[36521]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:25.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:25 smithi153 ceph-mon[36228]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:27.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:27 smithi178 ceph-mon[36521]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:27.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:27 smithi153 ceph-mon[36228]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:27.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:27 smithi106 ceph-mon[31965]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:29.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:29 smithi178 ceph-mon[36521]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:29.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:29 smithi153 ceph-mon[36228]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:29 smithi106 ceph-mon[31965]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:30.408 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:30 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T02:03:30.408 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:30 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:30.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:30 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T02:03:30.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:30 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:30.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:30 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T02:03:30.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:30 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:31.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:31 smithi153 ceph-mon[36228]: Deploying daemon osd.1 on smithi153 2024-03-28T02:03:31.616 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:31 smithi153 ceph-mon[36228]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:31.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:31 smithi178 ceph-mon[36521]: Deploying daemon osd.1 on smithi153 2024-03-28T02:03:31.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:31 smithi178 ceph-mon[36521]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:31 smithi106 ceph-mon[31965]: Deploying daemon osd.1 on smithi153 2024-03-28T02:03:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:31 smithi106 ceph-mon[31965]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:33.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:33 smithi153 ceph-mon[36228]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:33 smithi178 ceph-mon[36521]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:33 smithi106 ceph-mon[31965]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:35.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:35 smithi153 ceph-mon[36228]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:35.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:35 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:35.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:35 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:35.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:35 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:35.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:35 smithi178 ceph-mon[36521]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:35.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:35 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:35.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:35 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:35.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:35 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:35.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:35 smithi106 ceph-mon[31965]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:35.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:35 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:35.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:35 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:35.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:35 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:37.614 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:37 smithi153 ceph-mon[36228]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:37.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:37 smithi178 ceph-mon[36521]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:37.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:37 smithi106 ceph-mon[31965]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:38.295 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 1 on host 'smithi153' 2024-03-28T02:03:38.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.300+0000 7f0a517fa700 1 -- 172.21.15.153:0/748139198 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0a540fe820 con 0x7f0a400a2800 2024-03-28T02:03:38.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0a400a2800 msgr2=0x7f0a400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:38.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0a400a2800 0x7f0a400a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0a44005950 tx=0x7f0a440058e0 comp rx=0 tx=0).stop 2024-03-28T02:03:38.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a541010c0 msgr2=0x7f0a5419a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:38.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a541010c0 0x7f0a5419a5f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00d980 tx=0x7f0a3c00dc90 comp rx=0 tx=0).stop 2024-03-28T02:03:38.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 shutdown_connections 2024-03-28T02:03:38.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0a400a2800 0x7f0a400a4cc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:38.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a54102c60 0x7f0a541a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:38.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a541010c0 0x7f0a5419a5f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:38.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 --2- 172.21.15.153:0/748139198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a541022c0 0x7f0a5419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:38.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 >> 172.21.15.153:0/748139198 conn(0x7f0a540fc660 msgr2=0x7f0a540fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:03:38.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 shutdown_connections 2024-03-28T02:03:38.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:03:38.303+0000 7f0a5a30c700 1 -- 172.21.15.153:0/748139198 wait complete. 2024-03-28T02:03:38.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:38.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:38.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:38.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:38.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:38.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:38.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:38.916 DEBUG:teuthology.orchestra.run.smithi153:osd.1> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T02:03:38.918 INFO:tasks.cephadm:Deploying osd.2 on smithi178 with /dev/vg_nvme/lv_4... 2024-03-28T02:03:38.918 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T02:03:39.161 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:03:39.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:39 smithi153 ceph-mon[36228]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:39.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:39 smithi153 ceph-mon[36228]: from='osd.1 [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T02:03:39.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:39 smithi153 ceph-mon[36228]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T02:03:39.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:39 smithi178 ceph-mon[36521]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:39.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:39 smithi178 ceph-mon[36521]: from='osd.1 [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T02:03:39.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:39 smithi178 ceph-mon[36521]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T02:03:39.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:39 smithi106 ceph-mon[31965]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:39.845 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:39 smithi106 ceph-mon[31965]: from='osd.1 [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T02:03:39.845 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:39 smithi106 ceph-mon[31965]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T02:03:40.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:40 smithi153 ceph-mon[36228]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T02:03:40.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:40 smithi153 ceph-mon[36228]: from='osd.1 [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:03:40.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:40 smithi153 ceph-mon[36228]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T02:03:40.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:40 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:40.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:40 smithi153 ceph-mon[36228]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:03:40.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:40 smithi153 ceph-mon[36228]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:03:40.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:40 smithi106 ceph-mon[31965]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T02:03:40.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:40 smithi106 ceph-mon[31965]: from='osd.1 [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:03:40.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:40 smithi106 ceph-mon[31965]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T02:03:40.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:40 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:40.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:40 smithi106 ceph-mon[31965]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:03:40.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:40 smithi106 ceph-mon[31965]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:03:40.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:40 smithi178 ceph-mon[36521]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T02:03:40.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:40 smithi178 ceph-mon[36521]: from='osd.1 [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:03:40.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:40 smithi178 ceph-mon[36521]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T02:03:40.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:40 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:40.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:40 smithi178 ceph-mon[36521]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:03:40.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:40 smithi178 ceph-mon[36521]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:03:41.002 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T02:03:41.003 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T02:03:41.003 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-03-28T02:03:41.003 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-03-28T02:03:41.003 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00811018 s, 1.3 GB/s 2024-03-28T02:03:41.003 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-03-28T02:03:41.195 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:03:40 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1[40804]: 2024-03-28T02:03:40.862+0000 7f40ac00e700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T02:03:41.195 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:03:40 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1[40804]: 2024-03-28T02:03:40.865+0000 7f40a3df9700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:03:41.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:41 smithi178 ceph-mon[36521]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:41.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:41 smithi178 ceph-mon[36521]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T02:03:41.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:41.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:41.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:41 smithi178 ceph-mon[36521]: from='osd.1 ' entity='osd.1' 2024-03-28T02:03:41.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:41.590 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_4 2024-03-28T02:03:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:41 smithi153 ceph-mon[36228]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:41 smithi153 ceph-mon[36228]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T02:03:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:41 smithi153 ceph-mon[36228]: from='osd.1 ' entity='osd.1' 2024-03-28T02:03:41.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:41.832 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:03:41.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:41 smithi106 ceph-mon[31965]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:41.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:41 smithi106 ceph-mon[31965]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T02:03:41.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:41 smithi106 ceph-mon[31965]: from='osd.1 ' entity='osd.1' 2024-03-28T02:03:41.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:42.552 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: osd.1 [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] boot 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:42.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:03:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:03:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: osd.1 [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] boot 2024-03-28T02:03:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T02:03:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:03:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: osd.1 [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] boot 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:03:42.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:43.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 -- 172.21.15.178:0/2319601655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3064103d60 msgr2=0x7f306410a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:43.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 --2- 172.21.15.178:0/2319601655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3064103d60 0x7f306410a610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f305c00b0a0 tx=0x7f305c00b3b0 comp rx=0 tx=0).stop 2024-03-28T02:03:43.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 -- 172.21.15.178:0/2319601655 shutdown_connections 2024-03-28T02:03:43.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 --2- 172.21.15.178:0/2319601655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3064103d60 0x7f306410a610 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:43.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 --2- 172.21.15.178:0/2319601655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30641021c0 0x7f30641025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:43.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 --2- 172.21.15.178:0/2319601655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30641033c0 0x7f3064103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:43.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 -- 172.21.15.178:0/2319601655 >> 172.21.15.178:0/2319601655 conn(0x7f30640fd7a0 msgr2=0x7f30640ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:03:43.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 -- 172.21.15.178:0/2319601655 shutdown_connections 2024-03-28T02:03:43.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 -- 172.21.15.178:0/2319601655 wait complete. 2024-03-28T02:03:43.346 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 Processor -- start 2024-03-28T02:03:43.346 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.349+0000 7f306aa5d700 1 -- start start 2024-03-28T02:03:43.346 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f306aa5d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30641021c0 0x7f3064198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f306aa5d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30641033c0 0x7f30641988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f306aa5d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3064103d60 0x7f306419f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f306aa5d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306410ddc0 con 0x7f30641021c0 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f306aa5d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306410dc40 con 0x7f3064103d60 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f306aa5d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306410df40 con 0x7f30641033c0 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30637fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30641033c0 0x7f30641988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30637fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30641033c0 0x7f30641988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:39348/0 (socket says 172.21.15.178:39348) 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30637fe700 1 -- 172.21.15.178:0/3390337166 learned_addr learned my addr 172.21.15.178:0/3390337166 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f3063fff700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30641021c0 0x7f3064198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f3068ffa700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3064103d60 0x7f306419f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30637fe700 1 -- 172.21.15.178:0/3390337166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3064103d60 msgr2=0x7f306419f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30637fe700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3064103d60 0x7f306419f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:43.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30637fe700 1 -- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30641021c0 msgr2=0x7f3064198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30637fe700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30641021c0 0x7f3064198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30637fe700 1 -- 172.21.15.178:0/3390337166 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306419ffe0 con 0x7f30641033c0 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f3068ffa700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3064103d60 0x7f306419f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30637fe700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30641033c0 0x7f30641988c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f305800c950 tx=0x7f305800cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.350+0000 7f30617fa700 1 -- 172.21.15.178:0/3390337166 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058007ac0 con 0x7f30641033c0 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.351+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30641a02d0 con 0x7f30641033c0 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.351+0000 7f30617fa700 1 -- 172.21.15.178:0/3390337166 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3058004d10 con 0x7f30641033c0 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.351+0000 7f30617fa700 1 -- 172.21.15.178:0/3390337166 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305800f450 con 0x7f30641033c0 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.351+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30641a0810 con 0x7f30641033c0 2024-03-28T02:03:43.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.351+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3064068e50 con 0x7f30641033c0 2024-03-28T02:03:43.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.353+0000 7f30617fa700 1 -- 172.21.15.178:0/3390337166 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f3058004750 con 0x7f30641033c0 2024-03-28T02:03:43.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.353+0000 7f30617fa700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f304c0a2800 0x7f304c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:03:43.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.353+0000 7f30617fa700 1 -- 172.21.15.178:0/3390337166 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f30580cd100 con 0x7f30641033c0 2024-03-28T02:03:43.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.353+0000 7f3063fff700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f304c0a2800 0x7f304c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:03:43.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.354+0000 7f3063fff700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f304c0a2800 0x7f304c0a4cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3064103220 tx=0x7f305400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:03:43.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.357+0000 7f30617fa700 1 -- 172.21.15.178:0/3390337166 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3058014030 con 0x7f30641033c0 2024-03-28T02:03:43.558 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:03:43.563+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f30640625d0 con 0x7f304c0a2800 2024-03-28T02:03:43.559 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:43 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi153 2024-03-28T02:03:43.559 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:43 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi153 to 17080M 2024-03-28T02:03:43.560 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:43 smithi178 ceph-mon[36521]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:43 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi153 2024-03-28T02:03:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:43 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi153 to 17080M 2024-03-28T02:03:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:43 smithi153 ceph-mon[36228]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:43 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi153 2024-03-28T02:03:43.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:43 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi153 to 17080M 2024-03-28T02:03:43.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:43 smithi106 ceph-mon[31965]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T02:03:44.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:44 smithi153 ceph-mon[36228]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T02:03:44.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:03:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:03:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:44.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:44 smithi106 ceph-mon[31965]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T02:03:44.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:03:44.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:03:44.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:44.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:44 smithi178 ceph-mon[36521]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T02:03:44.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:03:44.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:03:44.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:45.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:45 smithi178 ceph-mon[36521]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:03:45.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:45 smithi178 ceph-mon[36521]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:45 smithi153 ceph-mon[36228]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:03:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:45 smithi153 ceph-mon[36228]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:45.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:45 smithi106 ceph-mon[31965]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:03:45.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:45 smithi106 ceph-mon[31965]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:46.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:46 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/2729250478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ba4df3-ae2f-4947-8eb9-e7d91e69bba2"}]: dispatch 2024-03-28T02:03:46.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:46 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ba4df3-ae2f-4947-8eb9-e7d91e69bba2"}]: dispatch 2024-03-28T02:03:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:46 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44ba4df3-ae2f-4947-8eb9-e7d91e69bba2"}]': finished 2024-03-28T02:03:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:46 smithi178 ceph-mon[36521]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T02:03:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:03:46.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:46 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/2729250478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ba4df3-ae2f-4947-8eb9-e7d91e69bba2"}]: dispatch 2024-03-28T02:03:46.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:46 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ba4df3-ae2f-4947-8eb9-e7d91e69bba2"}]: dispatch 2024-03-28T02:03:46.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:46 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44ba4df3-ae2f-4947-8eb9-e7d91e69bba2"}]': finished 2024-03-28T02:03:46.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:46 smithi153 ceph-mon[36228]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T02:03:46.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:03:46.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:46 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/2729250478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ba4df3-ae2f-4947-8eb9-e7d91e69bba2"}]: dispatch 2024-03-28T02:03:46.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:46 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ba4df3-ae2f-4947-8eb9-e7d91e69bba2"}]: dispatch 2024-03-28T02:03:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:46 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44ba4df3-ae2f-4947-8eb9-e7d91e69bba2"}]': finished 2024-03-28T02:03:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:46 smithi106 ceph-mon[31965]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T02:03:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:03:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:47 smithi153 ceph-mon[36228]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:47 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/1448958606' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:03:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:47 smithi106 ceph-mon[31965]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:47 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/1448958606' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:03:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:47 smithi178 ceph-mon[36521]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:47 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/1448958606' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:03:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:49 smithi153 ceph-mon[36228]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:49.743 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:49 smithi178 ceph-mon[36521]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:49 smithi106 ceph-mon[31965]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:51.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:51 smithi178 ceph-mon[36521]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:51 smithi153 ceph-mon[36228]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:51 smithi106 ceph-mon[31965]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:53.738 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:53 smithi178 ceph-mon[36521]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:53.739 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T02:03:53.739 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:53.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:53 smithi106 ceph-mon[31965]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:53.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T02:03:53.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:53.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:53 smithi153 ceph-mon[36228]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:53.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T02:03:53.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:03:54.549 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:54 smithi178 ceph-mon[36521]: Deploying daemon osd.2 on smithi178 2024-03-28T02:03:54.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:54 smithi153 ceph-mon[36228]: Deploying daemon osd.2 on smithi178 2024-03-28T02:03:54.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:54 smithi106 ceph-mon[31965]: Deploying daemon osd.2 on smithi178 2024-03-28T02:03:55.752 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:55 smithi153 ceph-mon[36228]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:55.798 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:55 smithi178 ceph-mon[36521]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:55.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:55 smithi106 ceph-mon[31965]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:57.815 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:57 smithi178 ceph-mon[36521]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:57.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:57 smithi106 ceph-mon[31965]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:57.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:57 smithi153 ceph-mon[36228]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:59.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:59 smithi178 ceph-mon[36521]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:59.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:59 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:59.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:59 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:59.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:03:59 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:59.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:59 smithi153 ceph-mon[36228]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:59.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:59 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:59.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:59 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:59.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:03:59 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:03:59.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:59 smithi106 ceph-mon[31965]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:03:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:59 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:59 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:03:59.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:03:59 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:01.481 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:01 smithi178 ceph-mon[36521]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:04:01.481 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:01 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:01.481 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:01 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:01.482 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:01 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:01.482 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:01 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:01.482 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:01 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:01.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:01 smithi153 ceph-mon[36228]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:04:01.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:01 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:01.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:01 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:01.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:01 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:01.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:01 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:01.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:01 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:01.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:01 smithi106 ceph-mon[31965]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:04:01.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:01 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:01.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:01 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:01.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:01 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:01.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:01 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:01.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:01 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:02.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.179+0000 7f30617fa700 1 -- 172.21.15.178:0/3390337166 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f30640625d0 con 0x7f304c0a2800 2024-03-28T02:04:02.173 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 2 on host 'smithi178' 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f304c0a2800 msgr2=0x7f304c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f304c0a2800 0x7f304c0a4cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3064103220 tx=0x7f305400b410 comp rx=0 tx=0).stop 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30641033c0 msgr2=0x7f30641988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30641033c0 0x7f30641988c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f305800c950 tx=0x7f305800cc60 comp rx=0 tx=0).stop 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 shutdown_connections 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3064103d60 0x7f306419f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f304c0a2800 0x7f304c0a4cc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30641021c0 0x7f3064198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 --2- 172.21.15.178:0/3390337166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30641033c0 0x7f30641988c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 >> 172.21.15.178:0/3390337166 conn(0x7f30640fd7a0 msgr2=0x7f3064106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:04:02.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 shutdown_connections 2024-03-28T02:04:02.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:02.182+0000 7f306aa5d700 1 -- 172.21.15.178:0/3390337166 wait complete. 2024-03-28T02:04:02.795 DEBUG:teuthology.orchestra.run.smithi178:osd.2> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T02:04:02.797 INFO:tasks.cephadm:Deploying osd.3 on smithi106 with /dev/vg_nvme/lv_3... 2024-03-28T02:04:02.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T02:04:03.052 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:04:03.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:03 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:03.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:03 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:03.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:03 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:03.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:03 smithi106 ceph-mon[31965]: from='osd.2 [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T02:04:03.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:03 smithi106 ceph-mon[31965]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T02:04:03.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:03 smithi106 ceph-mon[31965]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:04:03.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:03 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:03.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:03 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:03.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:03 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:03.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:03 smithi178 ceph-mon[36521]: from='osd.2 [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T02:04:03.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:03 smithi178 ceph-mon[36521]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T02:04:03.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:03 smithi178 ceph-mon[36521]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:04:03.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:03 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:03.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:03 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:03.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:03 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:03.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:03 smithi153 ceph-mon[36228]: from='osd.2 [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T02:04:03.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:03 smithi153 ceph-mon[36228]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T02:04:03.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:03 smithi153 ceph-mon[36228]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:04:04.194 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:04 smithi106 ceph-mon[31965]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T02:04:04.194 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:04 smithi106 ceph-mon[31965]: from='osd.2 [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:04:04.195 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:04 smithi106 ceph-mon[31965]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T02:04:04.195 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:04.195 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:04 smithi106 ceph-mon[31965]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:04:04.251 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:04 smithi178 ceph-mon[36521]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T02:04:04.251 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:04 smithi178 ceph-mon[36521]: from='osd.2 [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:04:04.251 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:04 smithi178 ceph-mon[36521]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T02:04:04.251 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:04.252 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:04 smithi178 ceph-mon[36521]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:04:04.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:04 smithi153 ceph-mon[36228]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T02:04:04.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:04 smithi153 ceph-mon[36228]: from='osd.2 [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:04:04.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:04 smithi153 ceph-mon[36228]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T02:04:04.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:04.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:04 smithi153 ceph-mon[36228]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:04:05.091 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:04 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2[41645]: 2024-03-28T02:04:04.639+0000 7fdbee0d9700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T02:04:05.092 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:04 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2[41645]: 2024-03-28T02:04:04.644+0000 7fdbe5ec4700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:04:05.096 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T02:04:05.096 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T02:04:05.096 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-28T02:04:05.096 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-28T02:04:05.096 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115531 s, 908 MB/s 2024-03-28T02:04:05.096 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-28T02:04:05.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:05 smithi178 ceph-mon[36521]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:04:05.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:05 smithi178 ceph-mon[36521]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T02:04:05.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:05 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:05.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:05 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:05.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:05 smithi178 ceph-mon[36521]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:04:05.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:05 smithi178 ceph-mon[36521]: from='osd.2 ' entity='osd.2' 2024-03-28T02:04:05.402 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:05 smithi106 ceph-mon[31965]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:04:05.402 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:05 smithi106 ceph-mon[31965]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T02:04:05.402 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:05 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:05.402 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:05 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:05.402 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:05 smithi106 ceph-mon[31965]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:04:05.402 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:05 smithi106 ceph-mon[31965]: from='osd.2 ' entity='osd.2' 2024-03-28T02:04:05.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:05 smithi153 ceph-mon[36228]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:04:05.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:05 smithi153 ceph-mon[36228]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T02:04:05.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:05 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:05.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:05 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:05.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:05 smithi153 ceph-mon[36228]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T02:04:05.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:05 smithi153 ceph-mon[36228]: from='osd.2 ' entity='osd.2' 2024-03-28T02:04:05.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_3 2024-03-28T02:04:05.979 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:04:06.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:04:06.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:04:06.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:06.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: osd.2 [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] boot 2024-03-28T02:04:06.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T02:04:06.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:06.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:06.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:06.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:06.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: osd.2 [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] boot 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:06.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:06.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: osd.2 [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] boot 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:06.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:06.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:06.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:07.243 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:07 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi178 2024-03-28T02:04:07.243 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:07 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi178 to 17080M 2024-03-28T02:04:07.243 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:07 smithi106 ceph-mon[31965]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:07.243 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:07 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T02:04:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:07 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi178 2024-03-28T02:04:07.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:07 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi178 to 17080M 2024-03-28T02:04:07.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:07 smithi153 ceph-mon[36228]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:07.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:07 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T02:04:07.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:07 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi178 2024-03-28T02:04:07.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:07 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi178 to 17080M 2024-03-28T02:04:07.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:07 smithi178 ceph-mon[36521]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:07.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:07 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T02:04:07.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1232918710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8107eb0 msgr2=0x7ffaf810a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:07.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1232918710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8107eb0 0x7ffaf810a320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000b0a0 tx=0x7ffaf000b3b0 comp rx=0 tx=0).stop 2024-03-28T02:04:07.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1232918710 shutdown_connections 2024-03-28T02:04:07.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1232918710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8107eb0 0x7ffaf810a320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:07.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1232918710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf8102c50 0x7ffaf8105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1232918710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf8105580 0x7ffaf8107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1232918710 >> 172.21.15.106:0/1232918710 conn(0x7ffaf80fc680 msgr2=0x7ffaf80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1232918710 shutdown_connections 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1232918710 wait complete. 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 Processor -- start 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.764+0000 7ffafe9a8700 1 -- start start 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffafe9a8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf8102c50 0x7ffaf81025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffafe9a8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8105580 0x7ffaf8100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffafe9a8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf8107eb0 0x7ffaf8101140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffafe9a8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf810dda0 con 0x7ffaf8105580 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffafe9a8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf810dc20 con 0x7ffaf8102c50 2024-03-28T02:04:07.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffafe9a8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf810df20 con 0x7ffaf8107eb0 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf77fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8105580 0x7ffaf8100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf77fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8105580 0x7ffaf8100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52584/0 (socket says 172.21.15.106:52584) 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf77fe700 1 -- 172.21.15.106:0/1762219683 learned_addr learned my addr 172.21.15.106:0/1762219683 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf7fff700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf8102c50 0x7ffaf81025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffafcf45700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf8107eb0 0x7ffaf8101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf77fe700 1 -- 172.21.15.106:0/1762219683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf8107eb0 msgr2=0x7ffaf8101140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf77fe700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf8107eb0 0x7ffaf8101140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf77fe700 1 -- 172.21.15.106:0/1762219683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf8102c50 msgr2=0x7ffaf81025b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf77fe700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf8102c50 0x7ffaf81025b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf77fe700 1 -- 172.21.15.106:0/1762219683 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf81019f0 con 0x7ffaf8105580 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf7fff700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf8102c50 0x7ffaf81025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.765+0000 7ffaf77fe700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8105580 0x7ffaf8100c00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00d980 tx=0x7ffaec00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:04:07.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.766+0000 7ffaf57fa700 1 -- 172.21.15.106:0/1762219683 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec017070 con 0x7ffaf8105580 2024-03-28T02:04:07.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.766+0000 7ffaf57fa700 1 -- 172.21.15.106:0/1762219683 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaec004d10 con 0x7ffaf8105580 2024-03-28T02:04:07.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.766+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf81a2470 con 0x7ffaf8105580 2024-03-28T02:04:07.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.766+0000 7ffaf57fa700 1 -- 172.21.15.106:0/1762219683 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec0055c0 con 0x7ffaf8105580 2024-03-28T02:04:07.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.766+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf81a2980 con 0x7ffaf8105580 2024-03-28T02:04:07.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.767+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf8068e50 con 0x7ffaf8105580 2024-03-28T02:04:07.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.769+0000 7ffaf57fa700 1 -- 172.21.15.106:0/1762219683 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7ffaec004750 con 0x7ffaf8105580 2024-03-28T02:04:07.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.769+0000 7ffaf57fa700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffae00a2880 0x7ffae00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:07.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.769+0000 7ffaf7fff700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffae00a2880 0x7ffae00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:07.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.769+0000 7ffaf57fa700 1 -- 172.21.15.106:0/1762219683 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7ffaec0cd890 con 0x7ffaf8105580 2024-03-28T02:04:07.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.770+0000 7ffaf7fff700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffae00a2880 0x7ffae00a4d40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffae8005fd0 tx=0x7ffae8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:04:07.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.773+0000 7ffaf57fa700 1 -- 172.21.15.106:0/1762219683 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffaec020020 con 0x7ffaf8105580 2024-03-28T02:04:07.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:07.976+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ffaf80625d0 con 0x7ffae00a2880 2024-03-28T02:04:08.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T02:04:08.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:08 smithi106 ceph-mon[31965]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T02:04:08.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T02:04:08.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:04:08.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:04:08.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:08.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T02:04:08.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:08 smithi153 ceph-mon[36228]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T02:04:08.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T02:04:08.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:04:08.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:04:08.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:08.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T02:04:08.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:08 smithi178 ceph-mon[36521]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T02:04:08.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T02:04:08.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:04:08.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:04:08.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:08.766 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:04:08 smithi106 sudo[51108]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T02:04:08.767 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:04:08 smithi106 sudo[51108]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:08.767 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:04:08 smithi106 sudo[51108]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:08.767 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:04:08 smithi106 sudo[51108]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:09.083 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:04:08 smithi106 sudo[51116]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T02:04:09.083 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:04:08 smithi106 sudo[51116]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:09.083 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:04:08 smithi106 sudo[51116]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:09.084 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:04:08 smithi106 sudo[51116]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:09.290 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:04:09 smithi153 sudo[44527]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T02:04:09.290 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:04:09 smithi153 sudo[44527]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:09.291 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:04:09 smithi153 sudo[44527]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:09.291 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:04:09 smithi153 sudo[44527]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:09 smithi153 ceph-mon[36228]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:04:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T02:04:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:09 smithi153 ceph-mon[36228]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T02:04:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:09 smithi153 ceph-mon[36228]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:09 smithi153 ceph-mon[36228]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T02:04:09.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:09 smithi178 ceph-mon[36521]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:04:09.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T02:04:09.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:09 smithi178 ceph-mon[36521]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T02:04:09.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:09 smithi178 ceph-mon[36521]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:09.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:09.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:09 smithi178 ceph-mon[36521]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T02:04:09.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:09 smithi106 ceph-mon[31965]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:04:09.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T02:04:09.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:09 smithi106 ceph-mon[31965]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T02:04:09.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:09 smithi106 ceph-mon[31965]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:09.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:09.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:09 smithi106 ceph-mon[31965]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T02:04:09.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:04:09 smithi153 sudo[44530]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T02:04:09.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:04:09 smithi153 sudo[44530]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:09.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:04:09 smithi153 sudo[44530]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:09.734 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:04:09 smithi153 sudo[44530]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:09.800 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:09.805+0000 7ffaf57fa700 1 -- 172.21.15.106:0/1762219683 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffaec097de0 con 0x7ffaf8105580 2024-03-28T02:04:09.812 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:09 smithi178 sudo[45730]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T02:04:09.812 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:09 smithi178 sudo[45730]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:09.812 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:09 smithi178 sudo[45730]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:09.812 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:09 smithi178 sudo[45730]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:10.122 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:09 smithi178 sudo[45733]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T02:04:10.123 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:09 smithi178 sudo[45733]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:10.123 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:09 smithi178 sudo[45733]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:10.123 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:04:09 smithi178 sudo[45733]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 sudo[51164]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T02:04:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 sudo[51164]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 sudo[51164]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 sudo[51164]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:10.607 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:10.607 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 ceph-mon[36521]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T02:04:10.607 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:10.607 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 ceph-mon[36521]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T02:04:10.607 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 sudo[45736]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T02:04:10.607 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 sudo[45736]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:10.607 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 sudo[45736]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:10.608 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 sudo[45736]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:10.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:10 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:10.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:10 smithi153 ceph-mon[36228]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T02:04:10.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:10 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:10.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:10 smithi153 ceph-mon[36228]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T02:04:10.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 sudo[51187]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T02:04:10.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 sudo[51187]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:10.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 sudo[51187]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:10.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 sudo[51187]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:10.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:10.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 ceph-mon[31965]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T02:04:10.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:10.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:10 smithi106 ceph-mon[31965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T02:04:10.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 sudo[45739]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T02:04:10.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 sudo[45739]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:10.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 sudo[45739]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:10.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:10 smithi178 sudo[45739]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:11.339 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 sudo[44533]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T02:04:11.339 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 sudo[44533]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:11.339 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 sudo[44533]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:11.339 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 sudo[44533]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3092977132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21a1eac9-0d56-4b09-914d-7a70444b90ed"}]: dispatch 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3092977132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21a1eac9-0d56-4b09-914d-7a70444b90ed"}]': finished 2024-03-28T02:04:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T02:04:11.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T02:04:11.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:11.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:04:11.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:04:11.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:04:11.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:11 smithi106 ceph-mon[31965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3092977132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21a1eac9-0d56-4b09-914d-7a70444b90ed"}]: dispatch 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T02:04:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3092977132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21a1eac9-0d56-4b09-914d-7a70444b90ed"}]': finished 2024-03-28T02:04:11.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T02:04:11.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T02:04:11.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:11.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:04:11.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:04:11.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:04:11.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:11 smithi178 ceph-mon[36521]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 sudo[44536]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 sudo[44536]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 sudo[44536]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 sudo[44536]: pam_unix(sudo:session): session closed for user root 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:04:11.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3092977132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21a1eac9-0d56-4b09-914d-7a70444b90ed"}]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3092977132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21a1eac9-0d56-4b09-914d-7a70444b90ed"}]': finished 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:04:11.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:11 smithi153 ceph-mon[36228]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T02:04:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:12 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/394209609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:04:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:12 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/394209609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:04:12.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:12 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/394209609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:04:13.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:13 smithi178 ceph-mon[36521]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:13.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:13 smithi153 ceph-mon[36228]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:13.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:13 smithi106 ceph-mon[31965]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:15.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:15 smithi106 ceph-mon[31965]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:15.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:15 smithi178 ceph-mon[36521]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:15.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:15 smithi153 ceph-mon[36228]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:17.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:17 smithi178 ceph-mon[36521]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:17 smithi153 ceph-mon[36228]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:17.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:17 smithi106 ceph-mon[31965]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:18.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:18 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T02:04:18.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:18 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:18.756 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:18 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T02:04:18.756 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:18 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:18 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T02:04:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:18 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:19 smithi153 ceph-mon[36228]: Deploying daemon osd.3 on smithi106 2024-03-28T02:04:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:19 smithi153 ceph-mon[36228]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:19.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:19 smithi106 ceph-mon[31965]: Deploying daemon osd.3 on smithi106 2024-03-28T02:04:19.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:19 smithi106 ceph-mon[31965]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:19 smithi178 ceph-mon[36521]: Deploying daemon osd.3 on smithi106 2024-03-28T02:04:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:19 smithi178 ceph-mon[36521]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:21.481 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:21 smithi106 ceph-mon[31965]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:21 smithi153 ceph-mon[36228]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:21 smithi178 ceph-mon[36521]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:23.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:23 smithi153 ceph-mon[36228]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:23.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:23 smithi106 ceph-mon[31965]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:23 smithi178 ceph-mon[36521]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:24 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:24 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:24 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:24 smithi153 ceph-mon[36228]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:25.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:24 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:25.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:24 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:25.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:24 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:25.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:24 smithi106 ceph-mon[31965]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:25.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:24 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:25.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:24 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:25.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:24 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:25.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:24 smithi178 ceph-mon[36521]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.383+0000 7ffaf57fa700 1 -- 172.21.15.106:0/1762219683 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffaf80625d0 con 0x7ffae00a2880 2024-03-28T02:04:27.378 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 3 on host 'smithi106' 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.385+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffae00a2880 msgr2=0x7ffae00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.385+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffae00a2880 0x7ffae00a4d40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffae8005fd0 tx=0x7ffae8005e30 comp rx=0 tx=0).stop 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.385+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8105580 msgr2=0x7ffaf8100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.385+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8105580 0x7ffaf8100c00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00d980 tx=0x7ffaec00dd40 comp rx=0 tx=0).stop 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.386+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 shutdown_connections 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.386+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf8107eb0 0x7ffaf8101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.386+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf8102c50 0x7ffaf81025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.386+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffae00a2880 0x7ffae00a4d40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.386+0000 7ffafe9a8700 1 --2- 172.21.15.106:0/1762219683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffaf8105580 0x7ffaf8100c00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.386+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 >> 172.21.15.106:0/1762219683 conn(0x7ffaf80fc680 msgr2=0x7ffaf8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.386+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 shutdown_connections 2024-03-28T02:04:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:04:27.386+0000 7ffafe9a8700 1 -- 172.21.15.106:0/1762219683 wait complete. 2024-03-28T02:04:27.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:27 smithi106 ceph-mon[31965]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:27 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:27 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:27 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:27 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:27 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:27 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:27 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:27 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:27 smithi153 ceph-mon[36228]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:27.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:27 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:27 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:27 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:27.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:27 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:27.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:27 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:27 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:27.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:27 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:27 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:27 smithi178 ceph-mon[36521]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:27.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:27 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:27 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:27 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:27.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:27 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:27.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:27 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:27 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:27.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:27 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:27.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:27 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:28.069 DEBUG:teuthology.orchestra.run.smithi106:osd.3> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T02:04:28.072 INFO:tasks.cephadm:Deploying osd.4 on smithi153 with /dev/vg_nvme/lv_3... 2024-03-28T02:04:28.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T02:04:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:04:28.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:28 smithi153 ceph-mon[36228]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T02:04:28.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:28 smithi106 ceph-mon[31965]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T02:04:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:28 smithi178 ceph-mon[36521]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T02:04:29.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:29 smithi106 ceph-mon[31965]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:29.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:29 smithi106 ceph-mon[31965]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T02:04:29.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:29 smithi106 ceph-mon[31965]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T02:04:29.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:29 smithi106 ceph-mon[31965]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:04:29.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:29.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:29 smithi153 ceph-mon[36228]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:29.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:29 smithi153 ceph-mon[36228]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T02:04:29.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:29 smithi153 ceph-mon[36228]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T02:04:29.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:29 smithi153 ceph-mon[36228]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:04:29.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:29 smithi178 ceph-mon[36521]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:29 smithi178 ceph-mon[36521]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T02:04:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:29 smithi178 ceph-mon[36521]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T02:04:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:29 smithi178 ceph-mon[36521]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:04:29.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.266 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T02:04:30.266 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T02:04:30.266 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-28T02:04:30.266 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-28T02:04:30.266 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116648 s, 899 MB/s 2024-03-28T02:04:30.266 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-28T02:04:30.333 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:04:29 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3[53254]: 2024-03-28T02:04:29.870+0000 7ffb1bd66700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T02:04:30.333 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:04:29 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3[53254]: 2024-03-28T02:04:29.877+0000 7ffb13b51700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:04:30.570 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:30 smithi153 ceph-mon[36228]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:04:30.570 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:30 smithi153 ceph-mon[36228]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T02:04:30.570 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:30 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.570 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:30 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.570 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:30 smithi153 ceph-mon[36228]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' 2024-03-28T02:04:30.570 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:30 smithi153 ceph-mon[36228]: osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] boot 2024-03-28T02:04:30.570 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:30 smithi153 ceph-mon[36228]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T02:04:30.570 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:30 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.649 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:30 smithi106 ceph-mon[31965]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:04:30.649 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:30 smithi106 ceph-mon[31965]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T02:04:30.649 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:30 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.649 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:30 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.649 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:30 smithi106 ceph-mon[31965]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' 2024-03-28T02:04:30.649 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:30 smithi106 ceph-mon[31965]: osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] boot 2024-03-28T02:04:30.649 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:30 smithi106 ceph-mon[31965]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T02:04:30.649 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:30 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:30 smithi178 ceph-mon[36521]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:04:30.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:30 smithi178 ceph-mon[36521]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T02:04:30.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:30 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:30 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:30 smithi178 ceph-mon[36521]: from='osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761]' entity='osd.3' 2024-03-28T02:04:30.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:30 smithi178 ceph-mon[36521]: osd.3 [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] boot 2024-03-28T02:04:30.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:30 smithi178 ceph-mon[36521]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T02:04:30.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:30 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:04:30.918 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-03-28T02:04:31.179 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:04:31.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:31.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:31 smithi153 ceph-mon[36228]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T02:04:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:31 smithi106 ceph-mon[31965]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:31.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:31 smithi178 ceph-mon[36521]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T02:04:32.650 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:32 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi106 2024-03-28T02:04:32.651 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:32 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi106 to 8544M 2024-03-28T02:04:32.651 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:32 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:32.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:32 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi106 2024-03-28T02:04:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:32 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi106 to 8544M 2024-03-28T02:04:32.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:32.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 -- 172.21.15.153:0/230847167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9381004d0 msgr2=0x7fd9381008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:32.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 --2- 172.21.15.153:0/230847167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9381004d0 0x7fd9381008f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd928009990 tx=0x7fd928009ca0 comp rx=0 tx=0).stop 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 -- 172.21.15.153:0/230847167 shutdown_connections 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 --2- 172.21.15.153:0/230847167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd938103e50 0x7fd93810a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 --2- 172.21.15.153:0/230847167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9381004d0 0x7fd9381008f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 --2- 172.21.15.153:0/230847167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9381034b0 0x7fd938103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 -- 172.21.15.153:0/230847167 >> 172.21.15.153:0/230847167 conn(0x7fd9380fbe30 msgr2=0x7fd9380fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 -- 172.21.15.153:0/230847167 shutdown_connections 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 -- 172.21.15.153:0/230847167 wait complete. 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.852+0000 7fd93cdec700 1 Processor -- start 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd93cdec700 1 -- start start 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd93cdec700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9381004d0 0x7fd9381983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd93cdec700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9381034b0 0x7fd938198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd93cdec700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd938103e50 0x7fd93819f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd93cdec700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93810dde0 con 0x7fd9381004d0 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd93cdec700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93810dc60 con 0x7fd938103e50 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd93cdec700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93810df60 con 0x7fd9381034b0 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd936d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd938103e50 0x7fd93819f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd936d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd938103e50 0x7fd93819f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49896/0 (socket says 172.21.15.153:49896) 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd936d9d700 1 -- 172.21.15.153:0/2621688912 learned_addr learned my addr 172.21.15.153:0/2621688912 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd93659c700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9381004d0 0x7fd9381983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd935d9b700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9381034b0 0x7fd938198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd936d9d700 1 -- 172.21.15.153:0/2621688912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9381034b0 msgr2=0x7fd938198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd936d9d700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9381034b0 0x7fd938198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:32.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd936d9d700 1 -- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9381004d0 msgr2=0x7fd9381983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd936d9d700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9381004d0 0x7fd9381983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd936d9d700 1 -- 172.21.15.153:0/2621688912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9381a0050 con 0x7fd938103e50 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd936d9d700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd938103e50 0x7fd93819f9b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd93000ada0 tx=0x7fd93000c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd9277fe700 1 -- 172.21.15.153:0/2621688912 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930017020 con 0x7fd938103e50 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd9277fe700 1 -- 172.21.15.153:0/2621688912 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd93000a0d0 con 0x7fd938103e50 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9381a0340 con 0x7fd938103e50 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.853+0000 7fd9277fe700 1 -- 172.21.15.153:0/2621688912 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930005480 con 0x7fd938103e50 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.854+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9381a0880 con 0x7fd938103e50 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.854+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938068e50 con 0x7fd938103e50 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.855+0000 7fd9277fe700 1 -- 172.21.15.153:0/2621688912 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd930005620 con 0x7fd938103e50 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.856+0000 7fd9277fe700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9200a2890 0x7fd9200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:32.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.856+0000 7fd9277fe700 1 -- 172.21.15.153:0/2621688912 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fd9300ccf80 con 0x7fd938103e50 2024-03-28T02:04:32.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.856+0000 7fd93659c700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9200a2890 0x7fd9200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:32.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.857+0000 7fd93659c700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9200a2890 0x7fd9200a4d50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd928003890 tx=0x7fd928005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:04:32.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:32.861+0000 7fd9277fe700 1 -- 172.21.15.153:0/2621688912 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd930021020 con 0x7fd938103e50 2024-03-28T02:04:32.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:32 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi106 2024-03-28T02:04:32.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:32 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi106 to 8544M 2024-03-28T02:04:32.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:33.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:33.064+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9380625d0 con 0x7fd9200a2890 2024-03-28T02:04:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:33 smithi153 ceph-mon[36228]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:33 smithi153 ceph-mon[36228]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T02:04:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:04:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:04:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:33 smithi153 ceph-mon[36228]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T02:04:33.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:33 smithi106 ceph-mon[31965]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:33.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:33 smithi106 ceph-mon[31965]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T02:04:33.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:04:33.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:04:33.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:33.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:33 smithi106 ceph-mon[31965]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T02:04:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:33 smithi178 ceph-mon[36521]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:33 smithi178 ceph-mon[36521]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T02:04:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:04:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:04:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:33 smithi178 ceph-mon[36521]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T02:04:34.490 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:34 smithi153 ceph-mon[36228]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:04:34.491 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:34 smithi153 ceph-mon[36228]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T02:04:34.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:34 smithi106 ceph-mon[31965]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:04:34.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:34 smithi106 ceph-mon[31965]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T02:04:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:34 smithi178 ceph-mon[36521]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:04:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:34 smithi178 ceph-mon[36521]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T02:04:35.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:35 smithi153 ceph-mon[36228]: pgmap v105: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:04:35.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:35 smithi153 ceph-mon[36228]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T02:04:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:35 smithi106 ceph-mon[31965]: pgmap v105: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:04:35.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:35 smithi106 ceph-mon[31965]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T02:04:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:35 smithi178 ceph-mon[36521]: pgmap v105: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:04:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:35 smithi178 ceph-mon[36521]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T02:04:36.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.153:0/3592003850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7fd91b-fc97-464b-bd5b-56e9bcf663fe"}]: dispatch 2024-03-28T02:04:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:36 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7fd91b-fc97-464b-bd5b-56e9bcf663fe"}]: dispatch 2024-03-28T02:04:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:36 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad7fd91b-fc97-464b-bd5b-56e9bcf663fe"}]': finished 2024-03-28T02:04:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:36 smithi153 ceph-mon[36228]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T02:04:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.153:0/546477041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:04:36.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:36 smithi106 ceph-mon[31965]: from='client.? 172.21.15.153:0/3592003850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7fd91b-fc97-464b-bd5b-56e9bcf663fe"}]: dispatch 2024-03-28T02:04:36.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:36 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7fd91b-fc97-464b-bd5b-56e9bcf663fe"}]: dispatch 2024-03-28T02:04:36.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:36 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad7fd91b-fc97-464b-bd5b-56e9bcf663fe"}]': finished 2024-03-28T02:04:36.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:36 smithi106 ceph-mon[31965]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T02:04:36.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:36 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:36.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:36 smithi106 ceph-mon[31965]: from='client.? 172.21.15.153:0/546477041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:04:36.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:36 smithi178 ceph-mon[36521]: from='client.? 172.21.15.153:0/3592003850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7fd91b-fc97-464b-bd5b-56e9bcf663fe"}]: dispatch 2024-03-28T02:04:36.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:36 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7fd91b-fc97-464b-bd5b-56e9bcf663fe"}]: dispatch 2024-03-28T02:04:36.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:36 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad7fd91b-fc97-464b-bd5b-56e9bcf663fe"}]': finished 2024-03-28T02:04:36.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:36 smithi178 ceph-mon[36521]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T02:04:36.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:36 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:36.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:36 smithi178 ceph-mon[36521]: from='client.? 172.21.15.153:0/546477041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:04:37.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:37 smithi153 ceph-mon[36228]: pgmap v107: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:04:37.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:37 smithi106 ceph-mon[31965]: pgmap v107: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:04:37.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:37 smithi178 ceph-mon[36521]: pgmap v107: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:04:38.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:38 smithi106 ceph-mon[31965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T02:04:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:38 smithi106 ceph-mon[31965]: Cluster is now healthy 2024-03-28T02:04:38.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:38 smithi178 ceph-mon[36521]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T02:04:38.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:38 smithi178 ceph-mon[36521]: Cluster is now healthy 2024-03-28T02:04:38.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:38 smithi153 ceph-mon[36228]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T02:04:38.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:38 smithi153 ceph-mon[36228]: Cluster is now healthy 2024-03-28T02:04:39.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:39 smithi106 ceph-mon[31965]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-28T02:04:39.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:39 smithi178 ceph-mon[36521]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-28T02:04:39.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:39 smithi153 ceph-mon[36228]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-28T02:04:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:41 smithi153 ceph-mon[36228]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-28T02:04:41.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:41 smithi106 ceph-mon[31965]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-28T02:04:41.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:41 smithi178 ceph-mon[36521]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-28T02:04:43.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:43 smithi153 ceph-mon[36228]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T02:04:43.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T02:04:43.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:43 smithi106 ceph-mon[31965]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T02:04:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T02:04:43.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:43 smithi178 ceph-mon[36521]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T02:04:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T02:04:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:44.671 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:44 smithi153 ceph-mon[36228]: Deploying daemon osd.4 on smithi153 2024-03-28T02:04:44.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:44 smithi106 ceph-mon[31965]: Deploying daemon osd.4 on smithi153 2024-03-28T02:04:44.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:44 smithi178 ceph-mon[36521]: Deploying daemon osd.4 on smithi153 2024-03-28T02:04:45.773 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:45 smithi178 ceph-mon[36521]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T02:04:45.818 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:45 smithi153 ceph-mon[36228]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T02:04:45.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:45 smithi106 ceph-mon[31965]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T02:04:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:47 smithi153 ceph-mon[36228]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-28T02:04:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:47 smithi106 ceph-mon[31965]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-28T02:04:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:47 smithi178 ceph-mon[36521]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-28T02:04:49.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:49.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:49.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:49.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:49 smithi106 ceph-mon[31965]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T02:04:49.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:49.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:49.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:49.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:49 smithi178 ceph-mon[36521]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T02:04:49.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:49.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:49.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:49.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:49 smithi153 ceph-mon[36228]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T02:04:51.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:51 smithi178 ceph-mon[36521]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:51.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:51 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:51.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:51 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:51.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:51 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:51.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:51 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:51.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:51 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:51 smithi153 ceph-mon[36228]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:51 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:51.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:51 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:51.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:51 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:51.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:51 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:51.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:51 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:51 smithi106 ceph-mon[31965]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:51 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:51 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:51.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:51 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:51.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:51 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:51.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:51 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:52.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.228+0000 7fd9277fe700 1 -- 172.21.15.153:0/2621688912 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd9380625d0 con 0x7fd9200a2890 2024-03-28T02:04:52.223 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 4 on host 'smithi153' 2024-03-28T02:04:52.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9200a2890 msgr2=0x7fd9200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:52.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9200a2890 0x7fd9200a4d50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd928003890 tx=0x7fd928005fd0 comp rx=0 tx=0).stop 2024-03-28T02:04:52.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd938103e50 msgr2=0x7fd93819f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:52.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd938103e50 0x7fd93819f9b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd93000ada0 tx=0x7fd93000c330 comp rx=0 tx=0).stop 2024-03-28T02:04:52.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 shutdown_connections 2024-03-28T02:04:52.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd938103e50 0x7fd93819f9b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:52.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9200a2890 0x7fd9200a4d50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:52.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9381004d0 0x7fd9381983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:52.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 --2- 172.21.15.153:0/2621688912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9381034b0 0x7fd938198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:52.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 >> 172.21.15.153:0/2621688912 conn(0x7fd9380fbe30 msgr2=0x7fd9381067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:04:52.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 shutdown_connections 2024-03-28T02:04:52.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:04:52.231+0000 7fd93cdec700 1 -- 172.21.15.153:0/2621688912 wait complete. 2024-03-28T02:04:52.825 DEBUG:teuthology.orchestra.run.smithi153:osd.4> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T02:04:52.827 INFO:tasks.cephadm:Deploying osd.5 on smithi178 with /dev/vg_nvme/lv_3... 2024-03-28T02:04:52.827 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T02:04:53.070 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:04:53.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:53.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:53.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:53 smithi153 ceph-mon[36228]: from='osd.4 [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T02:04:53.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:53 smithi153 ceph-mon[36228]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T02:04:53.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:53.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:53 smithi153 ceph-mon[36228]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:53.502 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:53.502 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:53.502 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:53 smithi178 ceph-mon[36521]: from='osd.4 [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T02:04:53.502 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:53 smithi178 ceph-mon[36521]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T02:04:53.502 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:53.502 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:53 smithi178 ceph-mon[36521]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:53.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:53.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:53.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:53 smithi106 ceph-mon[31965]: from='osd.4 [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T02:04:53.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:53 smithi106 ceph-mon[31965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T02:04:53.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:04:53.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:53 smithi106 ceph-mon[31965]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:54.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:54 smithi153 ceph-mon[36228]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T02:04:54.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:54 smithi153 ceph-mon[36228]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T02:04:54.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:54 smithi153 ceph-mon[36228]: from='osd.4 [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:04:54.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:54 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:54.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:54 smithi153 ceph-mon[36228]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:04:54.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:54 smithi178 ceph-mon[36521]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T02:04:54.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:54 smithi178 ceph-mon[36521]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T02:04:54.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:54 smithi178 ceph-mon[36521]: from='osd.4 [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:04:54.486 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:54 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:54.486 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:54 smithi178 ceph-mon[36521]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:04:54.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:54 smithi106 ceph-mon[31965]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T02:04:54.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:54 smithi106 ceph-mon[31965]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T02:04:54.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:54 smithi106 ceph-mon[31965]: from='osd.4 [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:04:54.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:54 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:54.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:54 smithi106 ceph-mon[31965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:04:54.733 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:04:54 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4[47346]: 2024-03-28T02:04:54.683+0000 7ff6858f4700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T02:04:54.734 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:04:54 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4[47346]: 2024-03-28T02:04:54.691+0000 7ff67d6df700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:04:55.036 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T02:04:55.037 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T02:04:55.037 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-03-28T02:04:55.037 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-03-28T02:04:55.037 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00836944 s, 1.3 GB/s 2024-03-28T02:04:55.037 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-03-28T02:04:55.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:55 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:04:55.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:55 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:04:55.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:55 smithi153 ceph-mon[36228]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:04:55.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:55 smithi153 ceph-mon[36228]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T02:04:55.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:55 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:55.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:55 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:55.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:55 smithi153 ceph-mon[36228]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:55.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:55 smithi153 ceph-mon[36228]: from='osd.4 ' entity='osd.4' 2024-03-28T02:04:55.490 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:55 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:04:55.491 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:55 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:04:55.491 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:55 smithi178 ceph-mon[36521]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:04:55.491 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:55 smithi178 ceph-mon[36521]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T02:04:55.491 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:55 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:55.491 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:55 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:55.491 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:55 smithi178 ceph-mon[36521]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:55.491 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:55 smithi178 ceph-mon[36521]: from='osd.4 ' entity='osd.4' 2024-03-28T02:04:55.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:55 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:04:55.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:55 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:04:55.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:55 smithi106 ceph-mon[31965]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:04:55.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:55 smithi106 ceph-mon[31965]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T02:04:55.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:55.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:55.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:55 smithi106 ceph-mon[31965]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T02:04:55.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:55 smithi106 ceph-mon[31965]: from='osd.4 ' entity='osd.4' 2024-03-28T02:04:55.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_3 2024-03-28T02:04:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:04:56.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:56.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: osd.4 [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] boot 2024-03-28T02:04:56.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T02:04:56.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:56.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:56.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:56.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.467 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:56.467 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:56.467 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:56.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: osd.4 [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] boot 2024-03-28T02:04:56.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T02:04:56.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:56.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:56.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:56.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:56.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:56.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: osd.4 [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] boot 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:04:56.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:04:57.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:57 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi153 2024-03-28T02:04:57.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:57 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi153 to 8540M 2024-03-28T02:04:57.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:57 smithi178 ceph-mon[36521]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:04:57.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:57 smithi178 ceph-mon[36521]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T02:04:57.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:57 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi153 2024-03-28T02:04:57.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:57 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi153 to 8540M 2024-03-28T02:04:57.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:57 smithi153 ceph-mon[36228]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:04:57.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:57 smithi153 ceph-mon[36228]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T02:04:57.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:57 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi153 2024-03-28T02:04:57.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:57 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi153 to 8540M 2024-03-28T02:04:57.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:57 smithi106 ceph-mon[31965]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:04:57.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:57 smithi106 ceph-mon[31965]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T02:04:57.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.729+0000 7efed1b8b700 1 -- 172.21.15.178:0/2683885869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 msgr2=0x7efecc103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:57.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.729+0000 7efed1b8b700 1 --2- 172.21.15.178:0/2683885869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 0x7efecc103810 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efec0009990 tx=0x7efec0009ca0 comp rx=0 tx=0).stop 2024-03-28T02:04:57.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.729+0000 7efed1b8b700 1 -- 172.21.15.178:0/2683885869 shutdown_connections 2024-03-28T02:04:57.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.729+0000 7efed1b8b700 1 --2- 172.21.15.178:0/2683885869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc103d50 0x7efecc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:57.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.729+0000 7efed1b8b700 1 --2- 172.21.15.178:0/2683885869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efecc1021b0 0x7efecc1025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:57.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.729+0000 7efed1b8b700 1 --2- 172.21.15.178:0/2683885869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 0x7efecc103810 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.729+0000 7efed1b8b700 1 -- 172.21.15.178:0/2683885869 >> 172.21.15.178:0/2683885869 conn(0x7efecc0fd770 msgr2=0x7efecc0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.729+0000 7efed1b8b700 1 -- 172.21.15.178:0/2683885869 shutdown_connections 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.729+0000 7efed1b8b700 1 -- 172.21.15.178:0/2683885869 wait complete. 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efed1b8b700 1 Processor -- start 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efed1b8b700 1 -- start start 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efed1b8b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efecc1021b0 0x7efecc1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efed1b8b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 0x7efecc198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efed1b8b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc103d50 0x7efecc19f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efed1b8b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc10dd80 con 0x7efecc103d50 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efed1b8b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc10dc00 con 0x7efecc1021b0 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efed1b8b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc10df00 con 0x7efecc1033b0 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efecaffd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 0x7efecc198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efecaffd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 0x7efecc198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:42634/0 (socket says 172.21.15.178:42634) 2024-03-28T02:04:57.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efecaffd700 1 -- 172.21.15.178:0/3727170827 learned_addr learned my addr 172.21.15.178:0/3727170827 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.730+0000 7efecb7fe700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efecc1021b0 0x7efecc1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efecbfff700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc103d50 0x7efecc19f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efecaffd700 1 -- 172.21.15.178:0/3727170827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efecc1021b0 msgr2=0x7efecc1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efecaffd700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efecc1021b0 0x7efecc1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efecaffd700 1 -- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc103d50 msgr2=0x7efecc19f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efecaffd700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc103d50 0x7efecc19f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efecaffd700 1 -- 172.21.15.178:0/3727170827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc1a0030 con 0x7efecc1033b0 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efecbfff700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc103d50 0x7efecc19f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efecaffd700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 0x7efecc198910 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efec00047b0 tx=0x7efec0009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efec8ff9700 1 -- 172.21.15.178:0/3727170827 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0027070 con 0x7efecc1033b0 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efec8ff9700 1 -- 172.21.15.178:0/3727170827 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efec0022470 con 0x7efecc1033b0 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efecc1a02c0 con 0x7efecc1033b0 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efec8ff9700 1 -- 172.21.15.178:0/3727170827 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000f6c0 con 0x7efecc1033b0 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.731+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efecc1a0770 con 0x7efecc1033b0 2024-03-28T02:04:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.732+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efecc068e50 con 0x7efecc1033b0 2024-03-28T02:04:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.733+0000 7efec8ff9700 1 -- 172.21.15.178:0/3727170827 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efec002a070 con 0x7efecc1033b0 2024-03-28T02:04:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.734+0000 7efec8ff9700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efeb40a2910 0x7efeb40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:04:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.734+0000 7efec8ff9700 1 -- 172.21.15.178:0/3727170827 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7efec00cee70 con 0x7efecc1033b0 2024-03-28T02:04:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.734+0000 7efecb7fe700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efeb40a2910 0x7efeb40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:04:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.734+0000 7efecb7fe700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efeb40a2910 0x7efeb40a4dd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efecc103210 tx=0x7efebc00c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:04:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.738+0000 7efec8ff9700 1 -- 172.21.15.178:0/3727170827 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efec0031070 con 0x7efecc1033b0 2024-03-28T02:04:57.932 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:04:57.938+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7efecc0625d0 con 0x7efeb40a2910 2024-03-28T02:04:58.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:58 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:04:58.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:58 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:04:58.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:58 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:58.531 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:58 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:04:58.531 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:58 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:04:58.531 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:58 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:58.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:04:58.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:04:58.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:04:59.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:59 smithi153 ceph-mon[36228]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:04:59.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:04:59 smithi153 ceph-mon[36228]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:04:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:59 smithi106 ceph-mon[31965]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:04:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:04:59 smithi106 ceph-mon[31965]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:04:59.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:59 smithi178 ceph-mon[36521]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:04:59.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:04:59 smithi178 ceph-mon[36521]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:01.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:01 smithi178 ceph-mon[36521]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:01.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:01 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/815788595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d9710de-3c2e-492e-8a73-e8483bc7b269"}]: dispatch 2024-03-28T02:05:01.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:01 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d9710de-3c2e-492e-8a73-e8483bc7b269"}]: dispatch 2024-03-28T02:05:01.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:01 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d9710de-3c2e-492e-8a73-e8483bc7b269"}]': finished 2024-03-28T02:05:01.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:01 smithi178 ceph-mon[36521]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T02:05:01.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:01 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:01.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:01 smithi153 ceph-mon[36228]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:01.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:01 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/815788595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d9710de-3c2e-492e-8a73-e8483bc7b269"}]: dispatch 2024-03-28T02:05:01.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:01 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d9710de-3c2e-492e-8a73-e8483bc7b269"}]: dispatch 2024-03-28T02:05:01.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:01 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d9710de-3c2e-492e-8a73-e8483bc7b269"}]': finished 2024-03-28T02:05:01.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:01 smithi153 ceph-mon[36228]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T02:05:01.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:01 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:01.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:01 smithi106 ceph-mon[31965]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:01.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:01 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/815788595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d9710de-3c2e-492e-8a73-e8483bc7b269"}]: dispatch 2024-03-28T02:05:01.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:01 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d9710de-3c2e-492e-8a73-e8483bc7b269"}]: dispatch 2024-03-28T02:05:01.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:01 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d9710de-3c2e-492e-8a73-e8483bc7b269"}]': finished 2024-03-28T02:05:01.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:01 smithi106 ceph-mon[31965]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T02:05:01.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:01 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:02.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/466224639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:05:02.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/466224639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:05:02.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/466224639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:05:03.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:03 smithi178 ceph-mon[36521]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:03.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:03 smithi153 ceph-mon[36228]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:03.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:03 smithi106 ceph-mon[31965]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:05.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:05 smithi178 ceph-mon[36521]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:05.639 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:05 smithi153 ceph-mon[36228]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:05.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:05 smithi106 ceph-mon[31965]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:07.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:07 smithi178 ceph-mon[36521]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:07.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:07 smithi153 ceph-mon[36228]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:07 smithi106 ceph-mon[31965]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:08.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T02:05:08.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:08.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T02:05:08.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T02:05:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:09.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:09 smithi178 ceph-mon[36521]: Deploying daemon osd.5 on smithi178 2024-03-28T02:05:09.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:09 smithi178 ceph-mon[36521]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:09.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:09 smithi153 ceph-mon[36228]: Deploying daemon osd.5 on smithi178 2024-03-28T02:05:09.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:09 smithi153 ceph-mon[36228]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:09 smithi106 ceph-mon[31965]: Deploying daemon osd.5 on smithi178 2024-03-28T02:05:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:09 smithi106 ceph-mon[31965]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:11.710 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:11 smithi178 ceph-mon[36521]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:11 smithi153 ceph-mon[36228]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:11.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:11 smithi106 ceph-mon[31965]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:13.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:13 smithi178 ceph-mon[36521]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:13.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:13 smithi153 ceph-mon[36228]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:13.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:13 smithi106 ceph-mon[31965]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:14.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:14.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:14.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:14.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:14.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:14.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:14.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:14.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:14.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:15.714 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:15 smithi178 ceph-mon[36521]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:15.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:15 smithi106 ceph-mon[31965]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:15.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:15 smithi153 ceph-mon[36228]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:17.297 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.303+0000 7efec8ff9700 1 -- 172.21.15.178:0/3727170827 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efecc0625d0 con 0x7efeb40a2910 2024-03-28T02:05:17.298 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 5 on host 'smithi178' 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efeb40a2910 msgr2=0x7efeb40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efeb40a2910 0x7efeb40a4dd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efecc103210 tx=0x7efebc00c040 comp rx=0 tx=0).stop 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 msgr2=0x7efecc198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 0x7efecc198910 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efec00047b0 tx=0x7efec0009f50 comp rx=0 tx=0).stop 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 shutdown_connections 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efeb40a2910 0x7efeb40a4dd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc103d50 0x7efecc19f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efecc1021b0 0x7efecc1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 --2- 172.21.15.178:0/3727170827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efecc1033b0 0x7efecc198910 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:17.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 >> 172.21.15.178:0/3727170827 conn(0x7efecc0fd770 msgr2=0x7efecc1066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:05:17.301 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 shutdown_connections 2024-03-28T02:05:17.301 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:05:17.306+0000 7efed1b8b700 1 -- 172.21.15.178:0/3727170827 wait complete. 2024-03-28T02:05:17.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:17.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:17.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:17.604 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.604 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:17.604 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.604 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.604 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='osd.5 [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T02:05:17.604 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:17 smithi178 ceph-mon[36521]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T02:05:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:17.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:17.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:17.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='osd.5 [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T02:05:17.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:17 smithi153 ceph-mon[36228]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:17.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='osd.5 [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T02:05:17.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:17 smithi106 ceph-mon[31965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T02:05:17.947 DEBUG:teuthology.orchestra.run.smithi178:osd.5> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T02:05:17.949 INFO:tasks.cephadm:Deploying osd.6 on smithi106 with /dev/vg_nvme/lv_2... 2024-03-28T02:05:17.949 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T02:05:18.203 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:05:19.511 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:19 smithi106 ceph-mon[31965]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T02:05:19.511 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:19 smithi106 ceph-mon[31965]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T02:05:19.511 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:19 smithi106 ceph-mon[31965]: from='osd.5 [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:05:19.511 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:19 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:19.511 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:19 smithi106 ceph-mon[31965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:05:19.511 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:19 smithi106 ceph-mon[31965]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:19.540 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:19 smithi178 ceph-mon[36521]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T02:05:19.541 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:19 smithi178 ceph-mon[36521]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T02:05:19.541 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:19 smithi178 ceph-mon[36521]: from='osd.5 [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:05:19.541 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:19 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:19.541 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:19 smithi178 ceph-mon[36521]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:05:19.541 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:19 smithi178 ceph-mon[36521]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:19 smithi153 ceph-mon[36228]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T02:05:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:19 smithi153 ceph-mon[36228]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T02:05:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:19 smithi153 ceph-mon[36228]: from='osd.5 [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:05:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:19 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:19.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:19 smithi153 ceph-mon[36228]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:05:19.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:19 smithi153 ceph-mon[36228]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:19.873 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:05:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:05:19.738+0000 7fa0bb728700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T02:05:19.873 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:05:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:05:19.745+0000 7fa0b3513700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:05:20.237 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T02:05:20.237 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T02:05:20.237 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-28T02:05:20.237 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-28T02:05:20.237 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113815 s, 921 MB/s 2024-03-28T02:05:20.237 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-28T02:05:20.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:20 smithi106 ceph-mon[31965]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:05:20.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:20 smithi106 ceph-mon[31965]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T02:05:20.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:20.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:20.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:20 smithi106 ceph-mon[31965]: from='osd.5 ' entity='osd.5' 2024-03-28T02:05:20.609 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:20 smithi178 ceph-mon[36521]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:05:20.609 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:20 smithi178 ceph-mon[36521]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T02:05:20.609 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:20.609 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:20.609 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:20 smithi178 ceph-mon[36521]: from='osd.5 ' entity='osd.5' 2024-03-28T02:05:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:20 smithi153 ceph-mon[36228]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:05:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:20 smithi153 ceph-mon[36228]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T02:05:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:20 smithi153 ceph-mon[36228]: from='osd.5 ' entity='osd.5' 2024-03-28T02:05:20.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_2 2024-03-28T02:05:21.140 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: osd.5 [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] boot 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:21.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:21.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:21.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:21.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: osd.5 [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] boot 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:21.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:05:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:05:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: osd.5 [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] boot 2024-03-28T02:05:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T02:05:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:05:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T02:05:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:22.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:22 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi178 2024-03-28T02:05:22.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:22 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi178 to 8540M 2024-03-28T02:05:22.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:22 smithi106 ceph-mon[31965]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T02:05:22.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:22 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi178 2024-03-28T02:05:22.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:22 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi178 to 8540M 2024-03-28T02:05:22.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:22 smithi178 ceph-mon[36521]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T02:05:22.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:22 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi178 2024-03-28T02:05:22.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:22 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi178 to 8540M 2024-03-28T02:05:22.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:22 smithi153 ceph-mon[36228]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T02:05:22.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3169760675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 msgr2=0x7f7bd81050c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:22.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3169760675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 0x7f7bd81050c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009990 tx=0x7f7bc8009ca0 comp rx=0 tx=0).stop 2024-03-28T02:05:22.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3169760675 shutdown_connections 2024-03-28T02:05:22.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3169760675 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bd8107f30 0x7f7bd810a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:22.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3169760675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 0x7f7bd81050c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:22.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3169760675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd8105600 0x7f7bd81079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:22.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3169760675 >> 172.21.15.106:0/3169760675 conn(0x7f7bd80fc660 msgr2=0x7f7bd80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:05:22.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3169760675 shutdown_connections 2024-03-28T02:05:22.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3169760675 wait complete. 2024-03-28T02:05:22.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.827+0000 7f7bde3c8700 1 Processor -- start 2024-03-28T02:05:22.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bde3c8700 1 -- start start 2024-03-28T02:05:22.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bde3c8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 0x7f7bd8102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:05:22.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bde3c8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bd8105600 0x7f7bd8100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:05:22.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bde3c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd8107f30 0x7f7bd81011c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:05:22.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bde3c8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd810de10 con 0x7f7bd8102cd0 2024-03-28T02:05:22.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bde3c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd810dc90 con 0x7f7bd8107f30 2024-03-28T02:05:22.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bde3c8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd810df90 con 0x7f7bd8105600 2024-03-28T02:05:22.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bd7fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 0x7f7bd8102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:05:22.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bd7fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 0x7f7bd8102630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34630/0 (socket says 172.21.15.106:34630) 2024-03-28T02:05:22.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bd7fff700 1 -- 172.21.15.106:0/3394933435 learned_addr learned my addr 172.21.15.106:0/3394933435 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bd77fe700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bd8105600 0x7f7bd8100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bdc965700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd8107f30 0x7f7bd81011c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bd7fff700 1 -- 172.21.15.106:0/3394933435 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bd8105600 msgr2=0x7f7bd8100c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bd7fff700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bd8105600 0x7f7bd8100c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bd7fff700 1 -- 172.21.15.106:0/3394933435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd8107f30 msgr2=0x7f7bd81011c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bd7fff700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd8107f30 0x7f7bd81011c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bd7fff700 1 -- 172.21.15.106:0/3394933435 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8101a70 con 0x7f7bd8102cd0 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.828+0000 7f7bdc965700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd8107f30 0x7f7bd81011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.829+0000 7f7bd7fff700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 0x7f7bd8102630 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800b5c0 tx=0x7f7bc8009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.829+0000 7f7bd57fa700 1 -- 172.21.15.106:0/3394933435 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8026070 con 0x7f7bd8102cd0 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.829+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd81a25b0 con 0x7f7bd8102cd0 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.829+0000 7f7bd57fa700 1 -- 172.21.15.106:0/3394933435 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bc8004150 con 0x7f7bd8102cd0 2024-03-28T02:05:22.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.829+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd81a2a80 con 0x7f7bd8102cd0 2024-03-28T02:05:22.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.829+0000 7f7bd57fa700 1 -- 172.21.15.106:0/3394933435 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8004bd0 con 0x7f7bd8102cd0 2024-03-28T02:05:22.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.830+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc4005360 con 0x7f7bd8102cd0 2024-03-28T02:05:22.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.831+0000 7f7bd57fa700 1 -- 172.21.15.106:0/3394933435 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7bc8029070 con 0x7f7bd8102cd0 2024-03-28T02:05:22.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.831+0000 7f7bd57fa700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7bc00a2910 0x7f7bc00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:05:22.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.832+0000 7f7bd77fe700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7bc00a2910 0x7f7bc00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:05:22.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.832+0000 7f7bd57fa700 1 -- 172.21.15.106:0/3394933435 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f7bc80ce520 con 0x7f7bd8102cd0 2024-03-28T02:05:22.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.832+0000 7f7bd77fe700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7bc00a2910 0x7f7bc00a4dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc005950 tx=0x7f7bcc00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:05:22.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:22.836+0000 7f7bd57fa700 1 -- 172.21.15.106:0/3394933435 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7bc805d740 con 0x7f7bd8102cd0 2024-03-28T02:05:23.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:23.040+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bc4000bf0 con 0x7f7bc00a2910 2024-03-28T02:05:23.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:23 smithi106 ceph-mon[31965]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T02:05:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:23 smithi106 ceph-mon[31965]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:05:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:05:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:23.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:23 smithi106 ceph-mon[31965]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T02:05:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:23 smithi178 ceph-mon[36521]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T02:05:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:23 smithi178 ceph-mon[36521]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:05:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:05:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:23 smithi178 ceph-mon[36521]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T02:05:23.634 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:23 smithi153 ceph-mon[36228]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T02:05:23.635 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:23 smithi153 ceph-mon[36228]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:23.635 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:05:23.635 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:05:23.635 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:23.635 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:23 smithi153 ceph-mon[36228]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T02:05:24.551 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:24 smithi106 ceph-mon[31965]: from='client.14343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:05:24.551 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:24 smithi106 ceph-mon[31965]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T02:05:24.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:24 smithi178 ceph-mon[36521]: from='client.14343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:05:24.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:24 smithi178 ceph-mon[36521]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T02:05:24.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:24 smithi153 ceph-mon[36228]: from='client.14343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:05:24.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:24 smithi153 ceph-mon[36228]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T02:05:25.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:25 smithi106 ceph-mon[31965]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T02:05:25.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:25 smithi178 ceph-mon[36521]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T02:05:25.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:25 smithi153 ceph-mon[36228]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T02:05:26.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:26 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/327060503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "063f0b59-1636-4cff-8de5-176afcb329b6"}]: dispatch 2024-03-28T02:05:26.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:26 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/327060503' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "063f0b59-1636-4cff-8de5-176afcb329b6"}]': finished 2024-03-28T02:05:26.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:26 smithi106 ceph-mon[31965]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T02:05:26.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:26.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:26 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/327060503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "063f0b59-1636-4cff-8de5-176afcb329b6"}]: dispatch 2024-03-28T02:05:26.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:26 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/327060503' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "063f0b59-1636-4cff-8de5-176afcb329b6"}]': finished 2024-03-28T02:05:26.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:26 smithi178 ceph-mon[36521]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T02:05:26.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:26.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:26 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/327060503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "063f0b59-1636-4cff-8de5-176afcb329b6"}]: dispatch 2024-03-28T02:05:26.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:26 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/327060503' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "063f0b59-1636-4cff-8de5-176afcb329b6"}]': finished 2024-03-28T02:05:26.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:26 smithi153 ceph-mon[36228]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T02:05:26.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:27.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:27 smithi106 ceph-mon[31965]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-28T02:05:27.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:27 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/129635492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:05:27.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:27 smithi178 ceph-mon[36521]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-28T02:05:27.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:27 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/129635492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:05:27.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:27 smithi153 ceph-mon[36228]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-28T02:05:27.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:27 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/129635492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:05:29.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:29 smithi106 ceph-mon[31965]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T02:05:29.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:29 smithi178 ceph-mon[36521]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T02:05:29.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:29 smithi153 ceph-mon[36228]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T02:05:31.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:31 smithi178 ceph-mon[36521]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:31.713 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:31 smithi106 ceph-mon[31965]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:31.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:31 smithi153 ceph-mon[36228]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:33.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:33 smithi106 ceph-mon[31965]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:33.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T02:05:33.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:33 smithi178 ceph-mon[36521]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T02:05:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:33 smithi153 ceph-mon[36228]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T02:05:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:34.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:34 smithi178 ceph-mon[36521]: Deploying daemon osd.6 on smithi106 2024-03-28T02:05:34.717 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:34 smithi106 ceph-mon[31965]: Deploying daemon osd.6 on smithi106 2024-03-28T02:05:34.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:34 smithi153 ceph-mon[36228]: Deploying daemon osd.6 on smithi106 2024-03-28T02:05:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:35 smithi106 ceph-mon[31965]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:35.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:35 smithi178 ceph-mon[36521]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:35.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:35 smithi153 ceph-mon[36228]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:37.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:37 smithi178 ceph-mon[36521]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:37.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:37 smithi153 ceph-mon[36228]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:37.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:37 smithi106 ceph-mon[31965]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:39.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:39 smithi106 ceph-mon[31965]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:39.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:39 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:39.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:39 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:39.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:39 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:39.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:39 smithi153 ceph-mon[36228]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:39.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:39 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:39.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:39 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:39.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:39 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:39.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:39 smithi178 ceph-mon[36521]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:39.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:39 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:39.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:39 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:39.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:39 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:41.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:41 smithi106 ceph-mon[31965]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:41 smithi153 ceph-mon[36228]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:41.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:41 smithi178 ceph-mon[36521]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:42.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.343+0000 7f7bd57fa700 1 -- 172.21.15.106:0/3394933435 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7bc4000bf0 con 0x7f7bc00a2910 2024-03-28T02:05:42.337 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 6 on host 'smithi106' 2024-03-28T02:05:42.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7bc00a2910 msgr2=0x7f7bc00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:42.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7bc00a2910 0x7f7bc00a4dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc005950 tx=0x7f7bcc00a300 comp rx=0 tx=0).stop 2024-03-28T02:05:42.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 msgr2=0x7f7bd8102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:42.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 0x7f7bd8102630 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800b5c0 tx=0x7f7bc8009f50 comp rx=0 tx=0).stop 2024-03-28T02:05:42.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 shutdown_connections 2024-03-28T02:05:42.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd8107f30 0x7f7bd81011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:42.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7bc00a2910 0x7f7bc00a4dd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:42.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd8102cd0 0x7f7bd8102630 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:42.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 --2- 172.21.15.106:0/3394933435 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bd8105600 0x7f7bd8100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:42.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 >> 172.21.15.106:0/3394933435 conn(0x7f7bd80fc660 msgr2=0x7f7bd8103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:05:42.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 shutdown_connections 2024-03-28T02:05:42.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:05:42.345+0000 7f7bde3c8700 1 -- 172.21.15.106:0/3394933435 wait complete. 2024-03-28T02:05:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:42.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:42.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:42 smithi178 ceph-mon[36521]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T02:05:42.982 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.982 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.982 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:42.982 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:42.982 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.982 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:42.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:42.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:42 smithi106 ceph-mon[31965]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T02:05:42.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:42.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:05:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:42 smithi153 ceph-mon[36228]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T02:05:43.007 DEBUG:teuthology.orchestra.run.smithi106:osd.6> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T02:05:43.009 INFO:tasks.cephadm:Deploying osd.7 on smithi153 with /dev/vg_nvme/lv_2... 2024-03-28T02:05:43.009 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T02:05:43.257 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:05:44.469 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:44 smithi106 ceph-mon[31965]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T02:05:44.469 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:44 smithi106 ceph-mon[31965]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T02:05:44.469 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:44 smithi106 ceph-mon[31965]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:05:44.469 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:44.469 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:05:44 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:05:44.340+0000 7f55d2540700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T02:05:44.469 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:05:44 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:05:44.350+0000 7f55ce333700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:05:44.610 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:44 smithi153 ceph-mon[36228]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T02:05:44.611 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:44 smithi153 ceph-mon[36228]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T02:05:44.611 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:44 smithi153 ceph-mon[36228]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:05:44.611 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:44.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:44 smithi178 ceph-mon[36521]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T02:05:44.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:44 smithi178 ceph-mon[36521]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T02:05:44.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:44 smithi178 ceph-mon[36521]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:05:44.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:45.461 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T02:05:45.461 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T02:05:45.461 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-28T02:05:45.461 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-28T02:05:45.461 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115502 s, 908 MB/s 2024-03-28T02:05:45.461 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-28T02:05:45.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:45 smithi153 ceph-mon[36228]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:45.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:45 smithi153 ceph-mon[36228]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:05:45.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:45 smithi153 ceph-mon[36228]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T02:05:45.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:45.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:45.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:45 smithi106 ceph-mon[31965]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:45 smithi106 ceph-mon[31965]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:05:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:45 smithi106 ceph-mon[31965]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T02:05:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:45.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:45.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:45 smithi178 ceph-mon[36521]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:45.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:45 smithi178 ceph-mon[36521]: from='osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:05:45.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:45 smithi178 ceph-mon[36521]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T02:05:45.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:45.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:46.195 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-03-28T02:05:46.437 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:05:46.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:05:46.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:05:46.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:46.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] boot 2024-03-28T02:05:46.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T02:05:46.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:46.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.468 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.468 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:46.468 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:46.468 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:46.468 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.468 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:46.468 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:46.468 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:05:46.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:05:46.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] boot 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:46.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: osd.6 [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] boot 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:46.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:05:46.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:05:47.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:47 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi106 2024-03-28T02:05:47.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:47 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi106 to 5696M 2024-03-28T02:05:47.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:47 smithi153 ceph-mon[36228]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:47.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:47 smithi153 ceph-mon[36228]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T02:05:47.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:47 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi106 2024-03-28T02:05:47.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:47 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi106 to 5696M 2024-03-28T02:05:47.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:47 smithi178 ceph-mon[36521]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:47.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:47 smithi178 ceph-mon[36521]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T02:05:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:47 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi106 2024-03-28T02:05:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:47 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi106 to 5696M 2024-03-28T02:05:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:47 smithi106 ceph-mon[31965]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T02:05:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:47 smithi106 ceph-mon[31965]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T02:05:48.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.105+0000 7f1d82a0c700 1 -- 172.21.15.153:0/2608300398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 msgr2=0x7f1d7c10a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.105+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/2608300398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 0x7f1d7c10a320 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1d7400b0a0 tx=0x7f1d7400b3b0 comp rx=0 tx=0).stop 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.106+0000 7f1d82a0c700 1 -- 172.21.15.153:0/2608300398 shutdown_connections 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.106+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/2608300398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 0x7f1d7c10a320 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.106+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/2608300398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d7c102c50 0x7f1d7c105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.106+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/2608300398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d7c105580 0x7f1d7c107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.106+0000 7f1d82a0c700 1 -- 172.21.15.153:0/2608300398 >> 172.21.15.153:0/2608300398 conn(0x7f1d7c0fc680 msgr2=0x7f1d7c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.106+0000 7f1d82a0c700 1 -- 172.21.15.153:0/2608300398 shutdown_connections 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.106+0000 7f1d82a0c700 1 -- 172.21.15.153:0/2608300398 wait complete. 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.106+0000 7f1d82a0c700 1 Processor -- start 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.106+0000 7f1d82a0c700 1 -- start start 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d82a0c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d7c102c50 0x7f1d7c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:05:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d82a0c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d7c105580 0x7f1d7c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d82a0c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 0x7f1d7c19f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d82a0c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c10ddd0 con 0x7f1d7c105580 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d82a0c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c10dc50 con 0x7f1d7c107eb0 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d82a0c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c10df50 con 0x7f1d7c102c50 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d80fa9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 0x7f1d7c19f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d80fa9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 0x7f1d7c19f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38342/0 (socket says 172.21.15.153:38342) 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d80fa9700 1 -- 172.21.15.153:0/741756828 learned_addr learned my addr 172.21.15.153:0/741756828 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d7bfff700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d7c102c50 0x7f1d7c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d7b7fe700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d7c105580 0x7f1d7c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d80fa9700 1 -- 172.21.15.153:0/741756828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d7c102c50 msgr2=0x7f1d7c198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d80fa9700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d7c102c50 0x7f1d7c198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d80fa9700 1 -- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d7c105580 msgr2=0x7f1d7c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:05:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d80fa9700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d7c105580 0x7f1d7c198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d80fa9700 1 -- 172.21.15.153:0/741756828 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c19ff90 con 0x7f1d7c107eb0 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d80fa9700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 0x7f1d7c19f8f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d74007f00 tx=0x7f1d74004d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d797fa700 1 -- 172.21.15.153:0/741756828 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74021070 con 0x7f1d7c107eb0 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c1a0220 con 0x7f1d7c107eb0 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d797fa700 1 -- 172.21.15.153:0/741756828 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d74007c40 con 0x7f1d7c107eb0 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d797fa700 1 -- 172.21.15.153:0/741756828 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d740053c0 con 0x7f1d7c107eb0 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.107+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c1a06d0 con 0x7f1d7c107eb0 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.109+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c068e50 con 0x7f1d7c107eb0 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.109+0000 7f1d797fa700 1 -- 172.21.15.153:0/741756828 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1d74024070 con 0x7f1d7c107eb0 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.110+0000 7f1d797fa700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1d640a2890 0x7f1d640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:05:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.110+0000 7f1d797fa700 1 -- 172.21.15.153:0/741756828 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f1d7400f020 con 0x7f1d7c107eb0 2024-03-28T02:05:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.110+0000 7f1d7bfff700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1d640a2890 0x7f1d640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:05:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.111+0000 7f1d7bfff700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1d640a2890 0x7f1d640a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c005950 tx=0x7f1d6c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:05:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.115+0000 7f1d797fa700 1 -- 172.21.15.153:0/741756828 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d7402b020 con 0x7f1d7c107eb0 2024-03-28T02:05:48.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:05:48.321+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d7c0625d0 con 0x7f1d640a2890 2024-03-28T02:05:48.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:48 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:05:48.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:48 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:05:48.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:48 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:48.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:48 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:05:48.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:48 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:05:48.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:48 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:48.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:48 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:05:48.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:48 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:05:48.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:48 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:49.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:49 smithi153 ceph-mon[36228]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:49.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:49 smithi153 ceph-mon[36228]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:05:49.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:49 smithi178 ceph-mon[36521]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:49.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:49 smithi178 ceph-mon[36521]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:05:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:49 smithi106 ceph-mon[31965]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:49 smithi106 ceph-mon[31965]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:05:51.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:51 smithi178 ceph-mon[36521]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:51.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:51 smithi178 ceph-mon[36521]: from='client.? 172.21.15.153:0/3726550516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f29280-5275-445e-93a1-c93e64f17496"}]: dispatch 2024-03-28T02:05:51.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:51 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f29280-5275-445e-93a1-c93e64f17496"}]: dispatch 2024-03-28T02:05:51.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:51 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f29280-5275-445e-93a1-c93e64f17496"}]': finished 2024-03-28T02:05:51.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:51 smithi178 ceph-mon[36521]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T02:05:51.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:51 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:05:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:51 smithi153 ceph-mon[36228]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:51 smithi153 ceph-mon[36228]: from='client.? 172.21.15.153:0/3726550516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f29280-5275-445e-93a1-c93e64f17496"}]: dispatch 2024-03-28T02:05:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:51 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f29280-5275-445e-93a1-c93e64f17496"}]: dispatch 2024-03-28T02:05:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:51 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f29280-5275-445e-93a1-c93e64f17496"}]': finished 2024-03-28T02:05:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:51 smithi153 ceph-mon[36228]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T02:05:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:51 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:05:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:51 smithi106 ceph-mon[31965]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:51 smithi106 ceph-mon[31965]: from='client.? 172.21.15.153:0/3726550516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f29280-5275-445e-93a1-c93e64f17496"}]: dispatch 2024-03-28T02:05:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:51 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f29280-5275-445e-93a1-c93e64f17496"}]: dispatch 2024-03-28T02:05:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:51 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f29280-5275-445e-93a1-c93e64f17496"}]': finished 2024-03-28T02:05:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:51 smithi106 ceph-mon[31965]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T02:05:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:51 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:05:52.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:52 smithi178 ceph-mon[36521]: from='client.? 172.21.15.153:0/2992545727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:05:52.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:52 smithi153 ceph-mon[36228]: from='client.? 172.21.15.153:0/2992545727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:05:52.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:52 smithi106 ceph-mon[31965]: from='client.? 172.21.15.153:0/2992545727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:05:53.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:53 smithi178 ceph-mon[36521]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:53.634 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:53 smithi153 ceph-mon[36228]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:53.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:53 smithi106 ceph-mon[31965]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:55.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:55 smithi178 ceph-mon[36521]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:55 smithi153 ceph-mon[36228]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:55.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:55 smithi106 ceph-mon[31965]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:57.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:57 smithi178 ceph-mon[36521]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:57.660 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:57 smithi153 ceph-mon[36228]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:57.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:57 smithi106 ceph-mon[31965]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:58.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:58 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T02:05:58.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:58 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:58.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T02:05:58.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:58.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:58 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T02:05:58.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:58 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:05:59.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:59 smithi153 ceph-mon[36228]: Deploying daemon osd.7 on smithi153 2024-03-28T02:05:59.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:05:59 smithi153 ceph-mon[36228]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:59.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:59 smithi106 ceph-mon[31965]: Deploying daemon osd.7 on smithi153 2024-03-28T02:05:59.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:05:59 smithi106 ceph-mon[31965]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:05:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:59 smithi178 ceph-mon[36521]: Deploying daemon osd.7 on smithi153 2024-03-28T02:05:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:05:59 smithi178 ceph-mon[36521]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:01 smithi153 ceph-mon[36228]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:01.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:01 smithi106 ceph-mon[31965]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:01.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:01 smithi178 ceph-mon[36521]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:03.635 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:03 smithi106 ceph-mon[31965]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:03.635 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:03 smithi153 ceph-mon[36228]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:03 smithi178 ceph-mon[36521]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:05.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:05.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:05.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:05.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:04 smithi153 ceph-mon[36228]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:05.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:05.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:05.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:05.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:04 smithi106 ceph-mon[31965]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:05.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:05.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:05.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:05.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:04 smithi178 ceph-mon[36521]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:07 smithi153 ceph-mon[36228]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:07 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:07 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:07 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:07 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:07.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:07 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:07.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:07 smithi178 ceph-mon[36521]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:07.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:07 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:07.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:07 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:07.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:07 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:07.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:07 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:07.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:07 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:07 smithi106 ceph-mon[31965]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:07 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:07 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:07 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:07 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:07.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:07 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.886+0000 7f1d797fa700 1 -- 172.21.15.153:0/741756828 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1d7c0625d0 con 0x7f1d640a2890 2024-03-28T02:06:07.880 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 7 on host 'smithi153' 2024-03-28T02:06:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1d640a2890 msgr2=0x7f1d640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1d640a2890 0x7f1d640a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c005950 tx=0x7f1d6c00b410 comp rx=0 tx=0).stop 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 msgr2=0x7f1d7c19f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 0x7f1d7c19f8f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d74007f00 tx=0x7f1d74004d70 comp rx=0 tx=0).stop 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 shutdown_connections 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d7c107eb0 0x7f1d7c19f8f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d7c102c50 0x7f1d7c198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1d640a2890 0x7f1d640a4d50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 --2- 172.21.15.153:0/741756828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d7c105580 0x7f1d7c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 >> 172.21.15.153:0/741756828 conn(0x7f1d7c0fc680 msgr2=0x7f1d7c1035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 shutdown_connections 2024-03-28T02:06:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:06:07.889+0000 7f1d82a0c700 1 -- 172.21.15.153:0/741756828 wait complete. 2024-03-28T02:06:08.580 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:08 smithi153 ceph-mon[36228]: from='osd.7 [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T02:06:08.580 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:08 smithi153 ceph-mon[36228]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T02:06:08.580 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:08.580 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:08.580 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:08 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:08.606 DEBUG:teuthology.orchestra.run.smithi153:osd.7> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T02:06:08.608 INFO:tasks.cephadm:Deploying osd.8 on smithi178 with /dev/vg_nvme/lv_2... 2024-03-28T02:06:08.608 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T02:06:08.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:08 smithi178 ceph-mon[36521]: from='osd.7 [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T02:06:08.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:08 smithi178 ceph-mon[36521]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T02:06:08.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:08.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:08.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:08 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:08 smithi106 ceph-mon[31965]: from='osd.7 [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T02:06:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:08 smithi106 ceph-mon[31965]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T02:06:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:08.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:08.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:08 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:08.854 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:06:09.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:09 smithi178 ceph-mon[36521]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:09.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:09 smithi178 ceph-mon[36521]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T02:06:09.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:09 smithi178 ceph-mon[36521]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T02:06:09.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:09.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:09 smithi178 ceph-mon[36521]: from='osd.7 [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:06:09.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:09 smithi178 ceph-mon[36521]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:06:09.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:09 smithi153 ceph-mon[36228]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:09.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:09 smithi153 ceph-mon[36228]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T02:06:09.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:09 smithi153 ceph-mon[36228]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T02:06:09.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:09.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:09 smithi153 ceph-mon[36228]: from='osd.7 [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:06:09.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:09 smithi153 ceph-mon[36228]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:06:09.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:06:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:06:09.386+0000 7f94dad59700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T02:06:09.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:06:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:06:09.393+0000 7f94d6b4c700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:06:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:09 smithi106 ceph-mon[31965]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:09 smithi106 ceph-mon[31965]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T02:06:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:09 smithi106 ceph-mon[31965]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T02:06:09.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:09.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:09 smithi106 ceph-mon[31965]: from='osd.7 [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:06:09.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:09 smithi106 ceph-mon[31965]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:06:10.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:10 smithi153 ceph-mon[36228]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:06:10.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:10 smithi153 ceph-mon[36228]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T02:06:10.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:10 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:10.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:10 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:10.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:10 smithi178 ceph-mon[36521]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:06:10.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:10 smithi178 ceph-mon[36521]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T02:06:10.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:10 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:10.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:10 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:10.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:10 smithi106 ceph-mon[31965]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:06:10.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:10 smithi106 ceph-mon[31965]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T02:06:10.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:10 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:10.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:10 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:10.933 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T02:06:10.933 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T02:06:10.933 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-03-28T02:06:10.933 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-03-28T02:06:10.933 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00829844 s, 1.3 GB/s 2024-03-28T02:06:10.933 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-03-28T02:06:11.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:11 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:06:11.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:11 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:06:11.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:11 smithi178 ceph-mon[36521]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:11.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:11 smithi178 ceph-mon[36521]: osd.7 [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] boot 2024-03-28T02:06:11.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:11 smithi178 ceph-mon[36521]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T02:06:11.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:11.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:11 smithi178 ceph-mon[36521]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T02:06:11.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_2 2024-03-28T02:06:11.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:11 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:06:11.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:11 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:06:11.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:11 smithi153 ceph-mon[36228]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:11.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:11 smithi153 ceph-mon[36228]: osd.7 [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] boot 2024-03-28T02:06:11.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:11 smithi153 ceph-mon[36228]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T02:06:11.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:11.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:11 smithi153 ceph-mon[36228]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T02:06:11.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:11 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:06:11.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:11 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:06:11.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:11 smithi106 ceph-mon[31965]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-28T02:06:11.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:11 smithi106 ceph-mon[31965]: osd.7 [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] boot 2024-03-28T02:06:11.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:11 smithi106 ceph-mon[31965]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T02:06:11.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:06:11.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:11 smithi106 ceph-mon[31965]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T02:06:11.867 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:06:12.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi153 2024-03-28T02:06:12.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi153 to 5693M 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:12.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:12 smithi178 ceph-mon[36521]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T02:06:13.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi153 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi153 to 5693M 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:13.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:12 smithi106 ceph-mon[31965]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T02:06:13.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi153 2024-03-28T02:06:13.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:13.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:13.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:13.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:13.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:13.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi153 to 5693M 2024-03-28T02:06:13.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:13.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:13.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:13.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:13.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:13.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:12 smithi153 ceph-mon[36228]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T02:06:13.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.387+0000 7fd31edaf700 1 -- 172.21.15.178:0/1106178217 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 msgr2=0x7fd3181014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.387+0000 7fd31edaf700 1 --2- 172.21.15.178:0/1106178217 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 0x7fd3181014e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009990 tx=0x7fd30c009ca0 comp rx=0 tx=0).stop 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.387+0000 7fd31edaf700 1 -- 172.21.15.178:0/1106178217 shutdown_connections 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.387+0000 7fd31edaf700 1 --2- 172.21.15.178:0/1106178217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd318102c60 0x7fd318109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.387+0000 7fd31edaf700 1 --2- 172.21.15.178:0/1106178217 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 0x7fd3181014e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.387+0000 7fd31edaf700 1 --2- 172.21.15.178:0/1106178217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3181022c0 0x7fd318102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.387+0000 7fd31edaf700 1 -- 172.21.15.178:0/1106178217 >> 172.21.15.178:0/1106178217 conn(0x7fd3180fc680 msgr2=0x7fd3180feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.387+0000 7fd31edaf700 1 -- 172.21.15.178:0/1106178217 shutdown_connections 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.387+0000 7fd31edaf700 1 -- 172.21.15.178:0/1106178217 wait complete. 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31edaf700 1 Processor -- start 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31edaf700 1 -- start start 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31edaf700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 0x7fd3181983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:06:13.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31edaf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3181022c0 0x7fd318198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31edaf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd318102c60 0x7fd31819f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31edaf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31810cd00 con 0x7fd318102c60 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31edaf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31810cb80 con 0x7fd3181022c0 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31edaf700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31810ce80 con 0x7fd3181010c0 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31cb4b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 0x7fd3181983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31cb4b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 0x7fd3181983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:50126/0 (socket says 172.21.15.178:50126) 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31cb4b700 1 -- 172.21.15.178:0/3350086569 learned_addr learned my addr 172.21.15.178:0/3350086569 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd317fff700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3181022c0 0x7fd318198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31d34c700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd318102c60 0x7fd31819f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31cb4b700 1 -- 172.21.15.178:0/3350086569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3181022c0 msgr2=0x7fd318198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31cb4b700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3181022c0 0x7fd318198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31cb4b700 1 -- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd318102c60 msgr2=0x7fd31819f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31cb4b700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd318102c60 0x7fd31819f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.388+0000 7fd31cb4b700 1 -- 172.21.15.178:0/3350086569 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3181a0030 con 0x7fd3181010c0 2024-03-28T02:06:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.389+0000 7fd31cb4b700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 0x7fd3181983d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd30c000c00 tx=0x7fd30c0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:06:13.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.389+0000 7fd315ffb700 1 -- 172.21.15.178:0/3350086569 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c027070 con 0x7fd3181010c0 2024-03-28T02:06:13.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.389+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3181a02c0 con 0x7fd3181010c0 2024-03-28T02:06:13.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.389+0000 7fd315ffb700 1 -- 172.21.15.178:0/3350086569 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd30c004150 con 0x7fd3181010c0 2024-03-28T02:06:13.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.389+0000 7fd315ffb700 1 -- 172.21.15.178:0/3350086569 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c00f460 con 0x7fd3181010c0 2024-03-28T02:06:13.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.389+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3181a07f0 con 0x7fd3181010c0 2024-03-28T02:06:13.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.390+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd318068e50 con 0x7fd3181010c0 2024-03-28T02:06:13.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.391+0000 7fd315ffb700 1 -- 172.21.15.178:0/3350086569 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd30c02a070 con 0x7fd3181010c0 2024-03-28T02:06:13.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.391+0000 7fd315ffb700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd3080a2890 0x7fd3080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:06:13.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.392+0000 7fd315ffb700 1 -- 172.21.15.178:0/3350086569 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5397+0+0 (secure 0 0 0) 0x7fd30c0cee00 con 0x7fd3181010c0 2024-03-28T02:06:13.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.392+0000 7fd317fff700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd3080a2890 0x7fd3080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:06:13.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.392+0000 7fd317fff700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd3080a2890 0x7fd3080a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd304009a10 tx=0x7fd304009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:06:13.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.396+0000 7fd315ffb700 1 -- 172.21.15.178:0/3350086569 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd30c013070 con 0x7fd3181010c0 2024-03-28T02:06:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:13.600+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3181a0a50 con 0x7fd3080a2890 2024-03-28T02:06:14.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:14 smithi153 ceph-mon[36228]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T02:06:14.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:06:14.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:06:14.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:14.532 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:14 smithi178 ceph-mon[36521]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T02:06:14.532 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:06:14.533 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:06:14.533 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:14.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:14 smithi106 ceph-mon[31965]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T02:06:14.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:06:14.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:06:14.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:15.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:15 smithi153 ceph-mon[36228]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:06:15.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:15 smithi153 ceph-mon[36228]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T02:06:15.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:15 smithi153 ceph-mon[36228]: pgmap v181: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:15.486 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:15 smithi178 ceph-mon[36521]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:06:15.486 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:15 smithi178 ceph-mon[36521]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T02:06:15.486 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:15 smithi178 ceph-mon[36521]: pgmap v181: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:15.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:15 smithi106 ceph-mon[31965]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:06:15.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:15 smithi106 ceph-mon[31965]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T02:06:15.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:15 smithi106 ceph-mon[31965]: pgmap v181: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:16.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:16 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/1775756798' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46ffe893-a42e-4374-a83a-1316c70cd49d"}]: dispatch 2024-03-28T02:06:16.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:16 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46ffe893-a42e-4374-a83a-1316c70cd49d"}]: dispatch 2024-03-28T02:06:16.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:16 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46ffe893-a42e-4374-a83a-1316c70cd49d"}]': finished 2024-03-28T02:06:16.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:16 smithi178 ceph-mon[36521]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T02:06:16.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:16 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:16 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/1775756798' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46ffe893-a42e-4374-a83a-1316c70cd49d"}]: dispatch 2024-03-28T02:06:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:16 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46ffe893-a42e-4374-a83a-1316c70cd49d"}]: dispatch 2024-03-28T02:06:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:16 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46ffe893-a42e-4374-a83a-1316c70cd49d"}]': finished 2024-03-28T02:06:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:16 smithi153 ceph-mon[36228]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T02:06:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:16 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:16.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:16 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/1775756798' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46ffe893-a42e-4374-a83a-1316c70cd49d"}]: dispatch 2024-03-28T02:06:16.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:16 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46ffe893-a42e-4374-a83a-1316c70cd49d"}]: dispatch 2024-03-28T02:06:16.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:16 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46ffe893-a42e-4374-a83a-1316c70cd49d"}]': finished 2024-03-28T02:06:16.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:16 smithi106 ceph-mon[31965]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T02:06:16.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:16 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:17 smithi153 ceph-mon[36228]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:17 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/3379830640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:06:17.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:17 smithi106 ceph-mon[31965]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:17.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:17 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/3379830640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:06:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:17 smithi178 ceph-mon[36521]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:17 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/3379830640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:06:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:19 smithi153 ceph-mon[36228]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:19.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:19 smithi106 ceph-mon[31965]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:19 smithi178 ceph-mon[36521]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:21.515 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:21 smithi178 ceph-mon[36521]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:21 smithi153 ceph-mon[36228]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:21 smithi106 ceph-mon[31965]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:23 smithi178 ceph-mon[36521]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:23.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:23 smithi153 ceph-mon[36228]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:23.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:23 smithi106 ceph-mon[31965]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:24.605 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:24 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T02:06:24.605 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:24 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:24.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:24 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T02:06:24.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:24 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:24.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:24 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T02:06:24.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:24 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:25.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:25 smithi178 ceph-mon[36521]: Deploying daemon osd.8 on smithi178 2024-03-28T02:06:25.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:25 smithi178 ceph-mon[36521]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:25.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:25 smithi153 ceph-mon[36228]: Deploying daemon osd.8 on smithi178 2024-03-28T02:06:25.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:25 smithi153 ceph-mon[36228]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:25.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:25 smithi106 ceph-mon[31965]: Deploying daemon osd.8 on smithi178 2024-03-28T02:06:25.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:25 smithi106 ceph-mon[31965]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:27.645 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:27 smithi178 ceph-mon[36521]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:27.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:27 smithi153 ceph-mon[36228]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:27.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:27 smithi106 ceph-mon[31965]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:29.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:29 smithi153 ceph-mon[36228]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:29.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:29.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:29.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:29.753 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:29 smithi178 ceph-mon[36521]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:29.753 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:29.754 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:29.754 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:29 smithi106 ceph-mon[31965]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:31.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:31 smithi178 ceph-mon[36521]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:31.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:31 smithi153 ceph-mon[36228]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:31 smithi106 ceph-mon[31965]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:33.260 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.265+0000 7fd315ffb700 1 -- 172.21.15.178:0/3350086569 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd3181a0a50 con 0x7fd3080a2890 2024-03-28T02:06:33.260 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 8 on host 'smithi178' 2024-03-28T02:06:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd3080a2890 msgr2=0x7fd3080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd3080a2890 0x7fd3080a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd304009a10 tx=0x7fd304009190 comp rx=0 tx=0).stop 2024-03-28T02:06:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 msgr2=0x7fd3181983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 0x7fd3181983d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd30c000c00 tx=0x7fd30c0057c0 comp rx=0 tx=0).stop 2024-03-28T02:06:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 shutdown_connections 2024-03-28T02:06:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd3080a2890 0x7fd3080a4d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd318102c60 0x7fd31819f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3181010c0 0x7fd3181983d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 --2- 172.21.15.178:0/3350086569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3181022c0 0x7fd318198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 >> 172.21.15.178:0/3350086569 conn(0x7fd3180fc680 msgr2=0x7fd3180fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:06:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 shutdown_connections 2024-03-28T02:06:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:06:33.269+0000 7fd31edaf700 1 -- 172.21.15.178:0/3350086569 wait complete. 2024-03-28T02:06:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:33.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:33.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='osd.8 [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T02:06:33.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:33 smithi178 ceph-mon[36521]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='osd.8 [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T02:06:33.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:33 smithi106 ceph-mon[31965]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T02:06:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='osd.8 [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T02:06:33.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:33 smithi153 ceph-mon[36228]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T02:06:33.983 DEBUG:teuthology.orchestra.run.smithi178:osd.8> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T02:06:33.986 INFO:tasks.cephadm:Deploying osd.9 on smithi106 with /dev/vg_nvme/lv_1... 2024-03-28T02:06:33.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T02:06:34.236 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:06:35.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:35 smithi153 ceph-mon[36228]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T02:06:35.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:35 smithi153 ceph-mon[36228]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T02:06:35.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:35 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:35.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:35 smithi153 ceph-mon[36228]: from='osd.8 [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:06:35.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:35 smithi153 ceph-mon[36228]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:06:35.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:35 smithi153 ceph-mon[36228]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:35.492 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:35 smithi178 ceph-mon[36521]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T02:06:35.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:35 smithi178 ceph-mon[36521]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T02:06:35.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:35 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:35.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:35 smithi178 ceph-mon[36521]: from='osd.8 [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:06:35.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:35 smithi178 ceph-mon[36521]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:06:35.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:35 smithi178 ceph-mon[36521]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:35.493 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:06:35 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:06:35.233+0000 7f6573283700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T02:06:35.493 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:06:35 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:06:35.243+0000 7f656f076700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:06:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:35 smithi106 ceph-mon[31965]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T02:06:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:35 smithi106 ceph-mon[31965]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T02:06:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:35 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:35 smithi106 ceph-mon[31965]: from='osd.8 [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:06:35.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:35 smithi106 ceph-mon[31965]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:06:35.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:35 smithi106 ceph-mon[31965]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:36.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:36 smithi106 ceph-mon[31965]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:06:36.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:36 smithi106 ceph-mon[31965]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T02:06:36.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:36 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:36.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:36 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:36.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:36 smithi153 ceph-mon[36228]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:06:36.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:36 smithi153 ceph-mon[36228]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T02:06:36.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:36.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:36 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:36.538 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T02:06:36.538 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T02:06:36.538 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-28T02:06:36.538 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-28T02:06:36.538 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114917 s, 912 MB/s 2024-03-28T02:06:36.538 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-28T02:06:36.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:36 smithi178 ceph-mon[36521]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:06:36.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:36 smithi178 ceph-mon[36521]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T02:06:36.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:36 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:36.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:36 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:37.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_1 2024-03-28T02:06:37.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:06:37.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:06:37.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:37.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: osd.8 [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] boot 2024-03-28T02:06:37.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:37.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:37 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:06:37.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:06:37.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:37.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: osd.8 [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] boot 2024-03-28T02:06:37.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T02:06:37.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:37.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:37 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.521 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:06:37.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: osd.8 [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] boot 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:37.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:06:37.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:37.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:37.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:37.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:37 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:38.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:38 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi178 2024-03-28T02:06:38.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:38 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi178 to 5693M 2024-03-28T02:06:38.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:38 smithi106 ceph-mon[31965]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T02:06:38.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:38 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi178 2024-03-28T02:06:38.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:38 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi178 to 5693M 2024-03-28T02:06:38.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:38 smithi178 ceph-mon[36521]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T02:06:38.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:38 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi178 2024-03-28T02:06:38.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:38 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi178 to 5693M 2024-03-28T02:06:38.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:38 smithi153 ceph-mon[36228]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T02:06:39.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 -- 172.21.15.106:0/3988353435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c1010c0 msgr2=0x7f103c1014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:39.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 --2- 172.21.15.106:0/3988353435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c1010c0 0x7f103c1014e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f10300099c0 tx=0x7f1030009cd0 comp rx=0 tx=0).stop 2024-03-28T02:06:39.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 -- 172.21.15.106:0/3988353435 shutdown_connections 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 --2- 172.21.15.106:0/3988353435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f103c102c60 0x7f103c109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 --2- 172.21.15.106:0/3988353435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c1010c0 0x7f103c1014e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 --2- 172.21.15.106:0/3988353435 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f103c1022c0 0x7f103c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 -- 172.21.15.106:0/3988353435 >> 172.21.15.106:0/3988353435 conn(0x7f103c0fc680 msgr2=0x7f103c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 -- 172.21.15.106:0/3988353435 shutdown_connections 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 -- 172.21.15.106:0/3988353435 wait complete. 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 Processor -- start 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 -- start start 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f103c1010c0 0x7f103c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f103c1022c0 0x7f103c19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c102c60 0x7f103c19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:06:39.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.176+0000 7f1042a90700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c10cc90 con 0x7f103c102c60 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f1042a90700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c10cb10 con 0x7f103c1022c0 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f1042a90700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c10ce10 con 0x7f103c1010c0 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104102d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c102c60 0x7f103c19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104102d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c102c60 0x7f103c19f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41594/0 (socket says 172.21.15.106:41594) 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104102d700 1 -- 172.21.15.106:0/4228790384 learned_addr learned my addr 172.21.15.106:0/4228790384 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f103bfff700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f103c1022c0 0x7f103c19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104082c700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f103c1010c0 0x7f103c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104102d700 1 -- 172.21.15.106:0/4228790384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f103c1010c0 msgr2=0x7f103c06b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104102d700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f103c1010c0 0x7f103c06b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104102d700 1 -- 172.21.15.106:0/4228790384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f103c1022c0 msgr2=0x7f103c19d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104102d700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f103c1022c0 0x7f103c19d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104102d700 1 -- 172.21.15.106:0/4228790384 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c19ff20 con 0x7f103c102c60 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104082c700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f103c1010c0 0x7f103c06b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f104102d700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c102c60 0x7f103c19f650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1034007d70 tx=0x7f103400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:06:39.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f1039ffb700 1 -- 172.21.15.106:0/4228790384 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034016020 con 0x7f103c102c60 2024-03-28T02:06:39.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f1039ffb700 1 -- 172.21.15.106:0/4228790384 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f103400ad40 con 0x7f103c102c60 2024-03-28T02:06:39.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103c1a0210 con 0x7f103c102c60 2024-03-28T02:06:39.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f1039ffb700 1 -- 172.21.15.106:0/4228790384 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034011600 con 0x7f103c102c60 2024-03-28T02:06:39.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.177+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c1a0700 con 0x7f103c102c60 2024-03-28T02:06:39.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.178+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103c068e50 con 0x7f103c102c60 2024-03-28T02:06:39.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.179+0000 7f1039ffb700 1 -- 172.21.15.106:0/4228790384 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1034019020 con 0x7f103c102c60 2024-03-28T02:06:39.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.180+0000 7f1039ffb700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f102c0a2890 0x7f102c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:06:39.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.180+0000 7f104082c700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f102c0a2890 0x7f102c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:06:39.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.180+0000 7f1039ffb700 1 -- 172.21.15.106:0/4228790384 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f10340ce090 con 0x7f103c102c60 2024-03-28T02:06:39.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.180+0000 7f104082c700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f102c0a2890 0x7f102c0a4d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f103000b580 tx=0x7f1030005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:06:39.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.185+0000 7f1039ffb700 1 -- 172.21.15.106:0/4228790384 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1034020020 con 0x7f103c102c60 2024-03-28T02:06:39.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:39.395+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f103c0625d0 con 0x7f102c0a2890 2024-03-28T02:06:39.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:39 smithi106 ceph-mon[31965]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T02:06:39.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:39 smithi106 ceph-mon[31965]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:39.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:39 smithi178 ceph-mon[36521]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T02:06:39.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:39 smithi178 ceph-mon[36521]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:39.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:39 smithi153 ceph-mon[36228]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T02:06:39.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:39 smithi153 ceph-mon[36228]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:40.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:40 smithi106 ceph-mon[31965]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T02:06:40.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:40 smithi106 ceph-mon[31965]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:06:40.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:40 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:06:40.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:40 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:06:40.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:40 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:40.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:40 smithi178 ceph-mon[36521]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T02:06:40.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:40 smithi178 ceph-mon[36521]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:06:40.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:40 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:06:40.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:40 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:06:40.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:40 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:40.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:40 smithi153 ceph-mon[36228]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T02:06:40.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:40 smithi153 ceph-mon[36228]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:06:40.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:40 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:06:40.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:40 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:06:40.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:40 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:41.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:41 smithi106 ceph-mon[31965]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T02:06:41.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:41 smithi106 ceph-mon[31965]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:41.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:41 smithi178 ceph-mon[36521]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T02:06:41.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:41 smithi178 ceph-mon[36521]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:41 smithi153 ceph-mon[36228]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T02:06:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:41 smithi153 ceph-mon[36228]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:42 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4254184206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16014c57-a839-460f-b9e5-e1a336f53c52"}]: dispatch 2024-03-28T02:06:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:42 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4254184206' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16014c57-a839-460f-b9e5-e1a336f53c52"}]': finished 2024-03-28T02:06:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:42 smithi106 ceph-mon[31965]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T02:06:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:06:42.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:42 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4254184206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16014c57-a839-460f-b9e5-e1a336f53c52"}]: dispatch 2024-03-28T02:06:42.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:42 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4254184206' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16014c57-a839-460f-b9e5-e1a336f53c52"}]': finished 2024-03-28T02:06:42.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:42 smithi178 ceph-mon[36521]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T02:06:42.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:06:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:42 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4254184206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16014c57-a839-460f-b9e5-e1a336f53c52"}]: dispatch 2024-03-28T02:06:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:42 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4254184206' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16014c57-a839-460f-b9e5-e1a336f53c52"}]': finished 2024-03-28T02:06:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:42 smithi153 ceph-mon[36228]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T02:06:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:06:43.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:43 smithi106 ceph-mon[31965]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-28T02:06:43.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:43 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3448358999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:06:43.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:43 smithi178 ceph-mon[36521]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-28T02:06:43.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:43 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3448358999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:06:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:43 smithi153 ceph-mon[36228]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-28T02:06:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:43 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3448358999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:06:45.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:45 smithi178 ceph-mon[36521]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T02:06:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:45 smithi153 ceph-mon[36228]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T02:06:45.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:45 smithi106 ceph-mon[31965]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T02:06:47.444 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:47 smithi106 ceph-mon[31965]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-28T02:06:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:47 smithi153 ceph-mon[36228]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-28T02:06:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:47 smithi178 ceph-mon[36521]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-28T02:06:49.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:49 smithi106 ceph-mon[31965]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T02:06:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:49 smithi153 ceph-mon[36228]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T02:06:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:49 smithi178 ceph-mon[36521]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T02:06:50.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T02:06:50.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:50 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:50.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T02:06:50.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:50 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:50.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T02:06:50.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:50 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:51.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:51 smithi106 ceph-mon[31965]: Deploying daemon osd.9 on smithi106 2024-03-28T02:06:51.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:51 smithi106 ceph-mon[31965]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T02:06:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:51 smithi153 ceph-mon[36228]: Deploying daemon osd.9 on smithi106 2024-03-28T02:06:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:51 smithi153 ceph-mon[36228]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T02:06:51.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:51 smithi178 ceph-mon[36521]: Deploying daemon osd.9 on smithi106 2024-03-28T02:06:51.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:51 smithi178 ceph-mon[36521]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T02:06:53.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:53 smithi106 ceph-mon[31965]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:53.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:53 smithi153 ceph-mon[36228]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:53.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:53 smithi178 ceph-mon[36521]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:55 smithi153 ceph-mon[36228]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:55.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:55 smithi106 ceph-mon[31965]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:55 smithi178 ceph-mon[36521]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:56.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:56.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:56.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:56.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:56 smithi106 ceph-mon[31965]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:56.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:56.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:56.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:56.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:56 smithi178 ceph-mon[36521]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:56.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:56.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:06:56.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:56.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:56 smithi153 ceph-mon[36228]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:59 smithi106 ceph-mon[31965]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:59 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:59 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:59 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:59 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:06:59 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:59.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:59 smithi153 ceph-mon[36228]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:59.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:59 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:59.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:59 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:59.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:59 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:59.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:59 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:59.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:06:59 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:59.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.856+0000 7f1039ffb700 1 -- 172.21.15.106:0/4228790384 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f103c0625d0 con 0x7f102c0a2890 2024-03-28T02:06:59.849 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 9 on host 'smithi106' 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f102c0a2890 msgr2=0x7f102c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f102c0a2890 0x7f102c0a4d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f103000b580 tx=0x7f1030005fb0 comp rx=0 tx=0).stop 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c102c60 msgr2=0x7f103c19f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c102c60 0x7f103c19f650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1034007d70 tx=0x7f103400a340 comp rx=0 tx=0).stop 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 shutdown_connections 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f102c0a2890 0x7f102c0a4d50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f103c102c60 0x7f103c19f650 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f103c1010c0 0x7f103c06b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 --2- 172.21.15.106:0/4228790384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f103c1022c0 0x7f103c19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 >> 172.21.15.106:0/4228790384 conn(0x7f103c0fc680 msgr2=0x7f103c1054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 shutdown_connections 2024-03-28T02:06:59.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:06:59.858+0000 7f1042a90700 1 -- 172.21.15.106:0/4228790384 wait complete. 2024-03-28T02:06:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:59 smithi178 ceph-mon[36521]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:06:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:59 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:59 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:06:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:59 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:06:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:59 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:06:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:06:59 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:00.508 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:00 smithi106 ceph-mon[31965]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T02:07:00.508 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:00 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:00.508 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:00 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:00.508 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:00 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:00.532 DEBUG:teuthology.orchestra.run.smithi106:osd.9> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T02:07:00.535 INFO:tasks.cephadm:Deploying osd.10 on smithi153 with /dev/vg_nvme/lv_1... 2024-03-28T02:07:00.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T02:07:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:00 smithi153 ceph-mon[36228]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T02:07:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:00 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:00 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:00 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:00.784 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:07:00.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:00 smithi178 ceph-mon[36521]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T02:07:00.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:00 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:00.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:00 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:00.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:00 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:01 smithi153 ceph-mon[36228]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:07:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:01 smithi153 ceph-mon[36228]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T02:07:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:01 smithi153 ceph-mon[36228]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T02:07:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:01 smithi153 ceph-mon[36228]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:07:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:01 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:01.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:01 smithi106 ceph-mon[31965]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:07:01.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:01 smithi106 ceph-mon[31965]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T02:07:01.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:01 smithi106 ceph-mon[31965]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T02:07:01.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:01 smithi106 ceph-mon[31965]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:07:01.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:01 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:01.767 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:07:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:07:01.393+0000 7f45aa187700 -1 osd.9 0 waiting for initial osdmap 2024-03-28T02:07:01.768 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:07:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:07:01.405+0000 7f45a5f7a700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:07:01.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:01 smithi178 ceph-mon[36521]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:07:01.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:01 smithi178 ceph-mon[36521]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T02:07:01.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:01 smithi178 ceph-mon[36521]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T02:07:01.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:01 smithi178 ceph-mon[36521]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-28T02:07:01.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:01 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:02.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:02 smithi153 ceph-mon[36228]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:07:02.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:02 smithi153 ceph-mon[36228]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T02:07:02.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:02 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:02.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:02 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:02.806 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T02:07:02.807 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T02:07:02.807 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-28T02:07:02.807 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-28T02:07:02.807 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156879 s, 668 MB/s 2024-03-28T02:07:02.807 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-28T02:07:02.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:02 smithi106 ceph-mon[31965]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:07:02.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:02 smithi106 ceph-mon[31965]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T02:07:02.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:02 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:02.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:02 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:02.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:02 smithi178 ceph-mon[36521]: from='osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-28T02:07:02.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:02 smithi178 ceph-mon[36521]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T02:07:02.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:02 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:02.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:02 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:03.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:03 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:07:03.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:03 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:07:03.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:03 smithi153 ceph-mon[36228]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:07:03.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:03 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:03.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:03 smithi153 ceph-mon[36228]: osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] boot 2024-03-28T02:07:03.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:03 smithi153 ceph-mon[36228]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T02:07:03.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:03 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:03.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-03-28T02:07:03.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:03 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:07:03.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:03 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:07:03.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:03 smithi106 ceph-mon[31965]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:07:03.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:03 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:03.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:03 smithi106 ceph-mon[31965]: osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] boot 2024-03-28T02:07:03.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:03 smithi106 ceph-mon[31965]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T02:07:03.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:03 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:07:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:03 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:07:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:03 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:07:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:03 smithi178 ceph-mon[36521]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T02:07:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:03 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:03 smithi178 ceph-mon[36521]: osd.9 [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] boot 2024-03-28T02:07:03.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:03 smithi178 ceph-mon[36521]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T02:07:03.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:03 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:07:04.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T02:07:04.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.649 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.649 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.649 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.649 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:04.649 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:04.649 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:04 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:04.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:04 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T02:07:04.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:04.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:04 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.560+0000 7f4214bea700 1 -- 172.21.15.153:0/804175582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 msgr2=0x7f42101014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.560+0000 7f4214bea700 1 --2- 172.21.15.153:0/804175582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 0x7f42101014e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f42000099c0 tx=0x7f4200009cd0 comp rx=0 tx=0).stop 2024-03-28T02:07:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.561+0000 7f4214bea700 1 -- 172.21.15.153:0/804175582 shutdown_connections 2024-03-28T02:07:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.561+0000 7f4214bea700 1 --2- 172.21.15.153:0/804175582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4210102c60 0x7f4210109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.561+0000 7f4214bea700 1 --2- 172.21.15.153:0/804175582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 0x7f42101014e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.561+0000 7f4214bea700 1 --2- 172.21.15.153:0/804175582 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42101022c0 0x7f4210102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.561+0000 7f4214bea700 1 -- 172.21.15.153:0/804175582 >> 172.21.15.153:0/804175582 conn(0x7f42100fc680 msgr2=0x7f42100feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:07:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.561+0000 7f4214bea700 1 -- 172.21.15.153:0/804175582 shutdown_connections 2024-03-28T02:07:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.561+0000 7f4214bea700 1 -- 172.21.15.153:0/804175582 wait complete. 2024-03-28T02:07:05.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.561+0000 7f4214bea700 1 Processor -- start 2024-03-28T02:07:05.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.561+0000 7f4214bea700 1 -- start start 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f4214bea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 0x7f421006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f4214bea700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42101022c0 0x7f421019d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f4214bea700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4210102c60 0x7f421019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f4214bea700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421010ccb0 con 0x7f4210102c60 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f4214bea700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421010cb30 con 0x7f42101010c0 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f4214bea700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421010ce30 con 0x7f42101022c0 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 0x7f421006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 0x7f421006b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40834/0 (socket says 172.21.15.153:40834) 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420e59c700 1 -- 172.21.15.153:0/2014426562 learned_addr learned my addr 172.21.15.153:0/2014426562 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420ed9d700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4210102c60 0x7f421019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420dd9b700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42101022c0 0x7f421019d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420ed9d700 1 -- 172.21.15.153:0/2014426562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42101022c0 msgr2=0x7f421019d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420ed9d700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42101022c0 0x7f421019d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420ed9d700 1 -- 172.21.15.153:0/2014426562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 msgr2=0x7f421006b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420ed9d700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 0x7f421006b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420ed9d700 1 -- 172.21.15.153:0/2014426562 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42101a0090 con 0x7f4210102c60 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420e59c700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 0x7f421006b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f420ed9d700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4210102c60 0x7f421019f7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f420800efc0 tx=0x7f420800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f41ff7fe700 1 -- 172.21.15.153:0/2014426562 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080092e0 con 0x7f4210102c60 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f41ff7fe700 1 -- 172.21.15.153:0/2014426562 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4208004bb0 con 0x7f4210102c60 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f41ff7fe700 1 -- 172.21.15.153:0/2014426562 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208004280 con 0x7f4210102c60 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42101a0320 con 0x7f4210102c60 2024-03-28T02:07:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.562+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42101a0860 con 0x7f4210102c60 2024-03-28T02:07:05.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.563+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4210068e50 con 0x7f4210102c60 2024-03-28T02:07:05.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.565+0000 7f41ff7fe700 1 -- 172.21.15.153:0/2014426562 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f420801b030 con 0x7f4210102c60 2024-03-28T02:07:05.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.565+0000 7f41ff7fe700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f41f80a2890 0x7f41f80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:05.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.565+0000 7f41ff7fe700 1 -- 172.21.15.153:0/2014426562 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f42080cded0 con 0x7f4210102c60 2024-03-28T02:07:05.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.565+0000 7f420e59c700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f41f80a2890 0x7f41f80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:05.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.566+0000 7f420e59c700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f41f80a2890 0x7f41f80a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4200000c00 tx=0x7f4200005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:07:05.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.569+0000 7f41ff7fe700 1 -- 172.21.15.153:0/2014426562 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4208022020 con 0x7f4210102c60 2024-03-28T02:07:05.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:05 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi106 2024-03-28T02:07:05.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:05 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi106 to 4272M 2024-03-28T02:07:05.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:05 smithi153 ceph-mon[36228]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:05.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:05.773+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f42100625d0 con 0x7f41f80a2890 2024-03-28T02:07:05.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:05 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi106 2024-03-28T02:07:05.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:05 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi106 to 4272M 2024-03-28T02:07:05.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:05 smithi106 ceph-mon[31965]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:05 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi106 2024-03-28T02:07:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:05 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi106 to 4272M 2024-03-28T02:07:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:05 smithi178 ceph-mon[36521]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:06.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:07:06.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:07:06.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:06 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:06.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:07:06.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:07:06.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:06 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:06.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:07:06.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:07:06.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:06 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:07.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:07 smithi153 ceph-mon[36228]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:07:07.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:07 smithi153 ceph-mon[36228]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:07.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:07 smithi106 ceph-mon[31965]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:07:07.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:07 smithi106 ceph-mon[31965]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:07.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:07 smithi178 ceph-mon[36521]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:07:07.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:07 smithi178 ceph-mon[36521]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:09.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:09 smithi153 ceph-mon[36228]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:09.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:09 smithi153 ceph-mon[36228]: from='client.? 172.21.15.153:0/1178293245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d65fed3-ba4f-4a3c-9469-c624012d5fa4"}]: dispatch 2024-03-28T02:07:09.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:09 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d65fed3-ba4f-4a3c-9469-c624012d5fa4"}]: dispatch 2024-03-28T02:07:09.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:09 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d65fed3-ba4f-4a3c-9469-c624012d5fa4"}]': finished 2024-03-28T02:07:09.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:09 smithi153 ceph-mon[36228]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T02:07:09.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:09.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:09 smithi153 ceph-mon[36228]: from='client.? 172.21.15.153:0/3684222592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:07:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:09 smithi106 ceph-mon[31965]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:09 smithi106 ceph-mon[31965]: from='client.? 172.21.15.153:0/1178293245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d65fed3-ba4f-4a3c-9469-c624012d5fa4"}]: dispatch 2024-03-28T02:07:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:09 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d65fed3-ba4f-4a3c-9469-c624012d5fa4"}]: dispatch 2024-03-28T02:07:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:09 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d65fed3-ba4f-4a3c-9469-c624012d5fa4"}]': finished 2024-03-28T02:07:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:09 smithi106 ceph-mon[31965]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T02:07:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:09.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:09 smithi106 ceph-mon[31965]: from='client.? 172.21.15.153:0/3684222592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:07:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:09 smithi178 ceph-mon[36521]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:09 smithi178 ceph-mon[36521]: from='client.? 172.21.15.153:0/1178293245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d65fed3-ba4f-4a3c-9469-c624012d5fa4"}]: dispatch 2024-03-28T02:07:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:09 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d65fed3-ba4f-4a3c-9469-c624012d5fa4"}]: dispatch 2024-03-28T02:07:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:09 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d65fed3-ba4f-4a3c-9469-c624012d5fa4"}]': finished 2024-03-28T02:07:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:09 smithi178 ceph-mon[36521]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T02:07:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:09 smithi178 ceph-mon[36521]: from='client.? 172.21.15.153:0/3684222592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:07:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:11 smithi153 ceph-mon[36228]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:11.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:11 smithi106 ceph-mon[31965]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:11.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:11 smithi178 ceph-mon[36521]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:13.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:13 smithi153 ceph-mon[36228]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:13.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:13 smithi106 ceph-mon[31965]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:13.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:13 smithi178 ceph-mon[36521]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:15.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:15 smithi178 ceph-mon[36521]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:15.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:15 smithi153 ceph-mon[36228]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:15.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:15 smithi106 ceph-mon[31965]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:16 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T02:07:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:16 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:16.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:16 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T02:07:16.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:16 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:16.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:16 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T02:07:16.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:16 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:17 smithi153 ceph-mon[36228]: Deploying daemon osd.10 on smithi153 2024-03-28T02:07:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:17 smithi153 ceph-mon[36228]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:17.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:17 smithi106 ceph-mon[31965]: Deploying daemon osd.10 on smithi153 2024-03-28T02:07:17.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:17 smithi106 ceph-mon[31965]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:17 smithi178 ceph-mon[36521]: Deploying daemon osd.10 on smithi153 2024-03-28T02:07:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:17 smithi178 ceph-mon[36521]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:19 smithi153 ceph-mon[36228]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:19.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:19 smithi106 ceph-mon[31965]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:19 smithi178 ceph-mon[36521]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:21.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:21 smithi153 ceph-mon[36228]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:21 smithi106 ceph-mon[31965]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:21 smithi178 ceph-mon[36521]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:23.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:23.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:23.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:23.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:23 smithi153 ceph-mon[36228]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:23.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:23.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:23.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:23.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:23 smithi106 ceph-mon[31965]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:23.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:23 smithi178 ceph-mon[36521]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:25.608 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:25 smithi153 ceph-mon[36228]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:25.682 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 10 on host 'smithi153' 2024-03-28T02:07:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.688+0000 7f41ff7fe700 1 -- 172.21.15.153:0/2014426562 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f42100625d0 con 0x7f41f80a2890 2024-03-28T02:07:25.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f41f80a2890 msgr2=0x7f41f80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f41f80a2890 0x7f41f80a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4200000c00 tx=0x7f4200005fb0 comp rx=0 tx=0).stop 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4210102c60 msgr2=0x7f421019f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4210102c60 0x7f421019f7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f420800efc0 tx=0x7f420800c3a0 comp rx=0 tx=0).stop 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 shutdown_connections 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f41f80a2890 0x7f41f80a4d50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4210102c60 0x7f421019f7c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42101010c0 0x7f421006b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 --2- 172.21.15.153:0/2014426562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42101022c0 0x7f421019d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 >> 172.21.15.153:0/2014426562 conn(0x7f42100fc680 msgr2=0x7f4210105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:07:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.691+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 shutdown_connections 2024-03-28T02:07:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-28T02:07:25.692+0000 7f4214bea700 1 -- 172.21.15.153:0/2014426562 wait complete. 2024-03-28T02:07:25.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:25 smithi106 ceph-mon[31965]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:25.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:25 smithi178 ceph-mon[36521]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:26.280 DEBUG:teuthology.orchestra.run.smithi153:osd.10> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T02:07:26.283 INFO:tasks.cephadm:Deploying osd.11 on smithi178 with /dev/vg_nvme/lv_1... 2024-03-28T02:07:26.283 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T02:07:26.536 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:07:26.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:26.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:26.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:26 smithi178 ceph-mon[36521]: from='osd.10 [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T02:07:26.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:26 smithi178 ceph-mon[36521]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T02:07:26.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:26 smithi178 ceph-mon[36521]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:26.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:26.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:26 smithi153 ceph-mon[36228]: from='osd.10 [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T02:07:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:26 smithi153 ceph-mon[36228]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T02:07:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:26 smithi153 ceph-mon[36228]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:27.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:27.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:27.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:26 smithi106 ceph-mon[31965]: from='osd.10 [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T02:07:27.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:26 smithi106 ceph-mon[31965]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T02:07:27.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:26 smithi106 ceph-mon[31965]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:27.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:27.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:27.966 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:27 smithi178 ceph-mon[36521]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T02:07:27.967 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:27 smithi178 ceph-mon[36521]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T02:07:27.967 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:27 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:27.967 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:27 smithi178 ceph-mon[36521]: from='osd.10 [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:07:27.967 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:27 smithi178 ceph-mon[36521]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:07:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:27 smithi153 ceph-mon[36228]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T02:07:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:27 smithi153 ceph-mon[36228]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T02:07:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:27 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:27 smithi153 ceph-mon[36228]: from='osd.10 [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:07:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:27 smithi153 ceph-mon[36228]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:07:28.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:27 smithi106 ceph-mon[31965]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T02:07:28.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:27 smithi106 ceph-mon[31965]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T02:07:28.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:27 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:28.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:27 smithi106 ceph-mon[31965]: from='osd.10 [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:07:28.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:27 smithi106 ceph-mon[31965]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-28T02:07:28.414 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:07:28.093+0000 7fe66d6da700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T02:07:28.415 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:07:28.108+0000 7fe6654c5700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:07:28.716 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T02:07:28.716 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T02:07:28.716 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-03-28T02:07:28.716 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-03-28T02:07:28.716 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00862268 s, 1.2 GB/s 2024-03-28T02:07:28.716 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-03-28T02:07:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:28 smithi178 ceph-mon[36521]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:07:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:28 smithi178 ceph-mon[36521]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T02:07:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:28 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:28 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:28 smithi178 ceph-mon[36521]: from='osd.10 ' entity='osd.10' 2024-03-28T02:07:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:28 smithi178 ceph-mon[36521]: osd.10 [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] boot 2024-03-28T02:07:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:28 smithi178 ceph-mon[36521]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T02:07:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:28 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:28.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:28 smithi178 ceph-mon[36521]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:28.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-mon[36228]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:07:28.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-mon[36228]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T02:07:28.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:28.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:28.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-mon[36228]: from='osd.10 ' entity='osd.10' 2024-03-28T02:07:28.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-mon[36228]: osd.10 [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] boot 2024-03-28T02:07:28.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-mon[36228]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T02:07:28.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:28.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:28 smithi153 ceph-mon[36228]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:29.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:28 smithi106 ceph-mon[31965]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-28T02:07:29.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:28 smithi106 ceph-mon[31965]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T02:07:29.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:28 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:29.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:28 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:29.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:28 smithi106 ceph-mon[31965]: from='osd.10 ' entity='osd.10' 2024-03-28T02:07:29.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:28 smithi106 ceph-mon[31965]: osd.10 [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] boot 2024-03-28T02:07:29.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:28 smithi106 ceph-mon[31965]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T02:07:29.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:28 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:07:29.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:28 smithi106 ceph-mon[31965]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T02:07:29.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_1 2024-03-28T02:07:29.776 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:07:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:29 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:07:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:29 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:07:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:29 smithi178 ceph-mon[36521]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T02:07:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:29 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:07:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:29 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:07:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:29 smithi153 ceph-mon[36228]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T02:07:30.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:29 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:07:30.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:29 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:07:30.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:29 smithi106 ceph-mon[31965]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T02:07:31.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi153 2024-03-28T02:07:31.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T02:07:31.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi153 to 4270M 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:31.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.534 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi153 2024-03-28T02:07:31.534 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T02:07:31.535 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.537 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi153 to 4270M 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:31.538 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:31.539 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.550+0000 7f275e870700 1 -- 172.21.15.178:0/2821855187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103460 msgr2=0x7f27581038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:31.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.550+0000 7f275e870700 1 --2- 172.21.15.178:0/2821855187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103460 0x7f27581038c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2748009990 tx=0x7f2748009ca0 comp rx=0 tx=0).stop 2024-03-28T02:07:31.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.550+0000 7f275e870700 1 -- 172.21.15.178:0/2821855187 shutdown_connections 2024-03-28T02:07:31.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.550+0000 7f275e870700 1 --2- 172.21.15.178:0/2821855187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2758103e00 0x7f275810a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:31.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.550+0000 7f275e870700 1 --2- 172.21.15.178:0/2821855187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2758100480 0x7f27581008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:31.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.550+0000 7f275e870700 1 --2- 172.21.15.178:0/2821855187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103460 0x7f27581038c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:31.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.550+0000 7f275e870700 1 -- 172.21.15.178:0/2821855187 >> 172.21.15.178:0/2821855187 conn(0x7f27580fbe00 msgr2=0x7f27580fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:07:31.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.550+0000 7f275e870700 1 -- 172.21.15.178:0/2821855187 shutdown_connections 2024-03-28T02:07:31.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.550+0000 7f275e870700 1 -- 172.21.15.178:0/2821855187 wait complete. 2024-03-28T02:07:31.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275e870700 1 Processor -- start 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275e870700 1 -- start start 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275e870700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2758100480 0x7f2758102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275e870700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2758103460 0x7f2758101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275e870700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103e00 0x7f2758101950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275e870700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275810de90 con 0x7f2758103460 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275e870700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275810dd10 con 0x7f2758100480 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275e870700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275810e010 con 0x7f2758103e00 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275ce0d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103e00 0x7f2758101950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275ce0d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103e00 0x7f2758101950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:56238/0 (socket says 172.21.15.178:56238) 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275ce0d700 1 -- 172.21.15.178:0/2928483969 learned_addr learned my addr 172.21.15.178:0/2928483969 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f27577fe700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2758103460 0x7f2758101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f2757fff700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2758100480 0x7f2758102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:31.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275ce0d700 1 -- 172.21.15.178:0/2928483969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2758100480 msgr2=0x7f2758102dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275ce0d700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2758100480 0x7f2758102dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275ce0d700 1 -- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2758103460 msgr2=0x7f2758101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275ce0d700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2758103460 0x7f2758101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f275ce0d700 1 -- 172.21.15.178:0/2928483969 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2758102200 con 0x7f2758103e00 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.551+0000 7f27577fe700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2758103460 0x7f2758101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.552+0000 7f275ce0d700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103e00 0x7f2758101950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f275000b970 tx=0x7f275000bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.552+0000 7f27557fa700 1 -- 172.21.15.178:0/2928483969 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2750019070 con 0x7f2758103e00 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.552+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27581a26b0 con 0x7f2758103e00 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.552+0000 7f27557fa700 1 -- 172.21.15.178:0/2928483969 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2750014440 con 0x7f2758103e00 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.552+0000 7f27557fa700 1 -- 172.21.15.178:0/2928483969 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27500075d0 con 0x7f2758103e00 2024-03-28T02:07:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.552+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27581a2c40 con 0x7f2758103e00 2024-03-28T02:07:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.553+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2758068e50 con 0x7f2758103e00 2024-03-28T02:07:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.554+0000 7f27557fa700 1 -- 172.21.15.178:0/2928483969 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f275001c070 con 0x7f2758103e00 2024-03-28T02:07:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.554+0000 7f27557fa700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27400a2890 0x7f27400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.555+0000 7f27557fa700 1 -- 172.21.15.178:0/2928483969 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f27500cdda0 con 0x7f2758103e00 2024-03-28T02:07:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.555+0000 7f2757fff700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27400a2890 0x7f27400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.555+0000 7f2757fff700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27400a2890 0x7f27400a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f274c006fd0 tx=0x7f274c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:07:31.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.558+0000 7f27557fa700 1 -- 172.21.15.178:0/2928483969 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f275009b470 con 0x7f2758103e00 2024-03-28T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi153 2024-03-28T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi153 to 4270M 2024-03-28T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:31.758 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:31.764+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f27580625d0 con 0x7f27400a2890 2024-03-28T02:07:32.509 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:07:32.509 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:07:32.509 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:32.509 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:32 smithi178 ceph-mon[36521]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T02:07:32.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:07:32.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:07:32.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:32.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:32 smithi106 ceph-mon[31965]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T02:07:32.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:32 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T02:07:32.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:32 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T02:07:32.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:32 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:32.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:32 smithi153 ceph-mon[36228]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T02:07:33.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:33 smithi106 ceph-mon[31965]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:07:33.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:33 smithi106 ceph-mon[31965]: pgmap v240: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:07:33.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:33 smithi106 ceph-mon[31965]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T02:07:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:33 smithi178 ceph-mon[36521]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:07:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:33 smithi178 ceph-mon[36521]: pgmap v240: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:07:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:33 smithi178 ceph-mon[36521]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T02:07:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:33 smithi153 ceph-mon[36228]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:07:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:33 smithi153 ceph-mon[36228]: pgmap v240: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:07:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:33 smithi153 ceph-mon[36228]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T02:07:34.499 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:34 smithi106 ceph-mon[31965]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T02:07:34.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:34 smithi178 ceph-mon[36521]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T02:07:34.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:34 smithi153 ceph-mon[36228]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T02:07:35.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:35 smithi178 ceph-mon[36521]: pgmap v242: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:07:35.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:35 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/1959359855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37920df5-7610-4acf-98e8-33fad58f4d2e"}]: dispatch 2024-03-28T02:07:35.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:35 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37920df5-7610-4acf-98e8-33fad58f4d2e"}]: dispatch 2024-03-28T02:07:35.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:35 smithi178 ceph-mon[36521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37920df5-7610-4acf-98e8-33fad58f4d2e"}]': finished 2024-03-28T02:07:35.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:35 smithi178 ceph-mon[36521]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T02:07:35.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:35 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:35 smithi106 ceph-mon[31965]: pgmap v242: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:07:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:35 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/1959359855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37920df5-7610-4acf-98e8-33fad58f4d2e"}]: dispatch 2024-03-28T02:07:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:35 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37920df5-7610-4acf-98e8-33fad58f4d2e"}]: dispatch 2024-03-28T02:07:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:35 smithi106 ceph-mon[31965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37920df5-7610-4acf-98e8-33fad58f4d2e"}]': finished 2024-03-28T02:07:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:35 smithi106 ceph-mon[31965]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T02:07:35.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:35 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:35.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:35 smithi153 ceph-mon[36228]: pgmap v242: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T02:07:35.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:35 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/1959359855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37920df5-7610-4acf-98e8-33fad58f4d2e"}]: dispatch 2024-03-28T02:07:35.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:35 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37920df5-7610-4acf-98e8-33fad58f4d2e"}]: dispatch 2024-03-28T02:07:35.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:35 smithi153 ceph-mon[36228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37920df5-7610-4acf-98e8-33fad58f4d2e"}]': finished 2024-03-28T02:07:35.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:35 smithi153 ceph-mon[36228]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T02:07:35.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:35 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:36.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:36 smithi106 ceph-mon[31965]: from='client.? 172.21.15.178:0/4162265900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:07:36.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:36 smithi178 ceph-mon[36521]: from='client.? 172.21.15.178:0/4162265900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:07:36.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.178:0/4162265900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T02:07:37.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:37 smithi106 ceph-mon[31965]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T02:07:37.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:37 smithi178 ceph-mon[36521]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T02:07:37.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:37 smithi153 ceph-mon[36228]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T02:07:38.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:38 smithi106 ceph-mon[31965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T02:07:38.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:38 smithi106 ceph-mon[31965]: Cluster is now healthy 2024-03-28T02:07:38.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:38 smithi178 ceph-mon[36521]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T02:07:38.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:38 smithi178 ceph-mon[36521]: Cluster is now healthy 2024-03-28T02:07:38.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:38 smithi153 ceph-mon[36228]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T02:07:38.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:38 smithi153 ceph-mon[36228]: Cluster is now healthy 2024-03-28T02:07:39.558 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:39 smithi178 ceph-mon[36521]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T02:07:39.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:39 smithi106 ceph-mon[31965]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T02:07:39.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:39 smithi153 ceph-mon[36228]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T02:07:41.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:41 smithi153 ceph-mon[36228]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T02:07:41.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:41 smithi106 ceph-mon[31965]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T02:07:41.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:41 smithi178 ceph-mon[36521]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T02:07:42.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T02:07:42.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T02:07:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:42.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T02:07:42.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:43.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:43 smithi178 ceph-mon[36521]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T02:07:43.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:43 smithi178 ceph-mon[36521]: Deploying daemon osd.11 on smithi178 2024-03-28T02:07:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:43 smithi153 ceph-mon[36228]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T02:07:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:43 smithi153 ceph-mon[36228]: Deploying daemon osd.11 on smithi178 2024-03-28T02:07:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:43 smithi106 ceph-mon[31965]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T02:07:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:43 smithi106 ceph-mon[31965]: Deploying daemon osd.11 on smithi178 2024-03-28T02:07:45.508 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:45 smithi178 ceph-mon[36521]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T02:07:45.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:45 smithi106 ceph-mon[31965]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T02:07:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:45 smithi153 ceph-mon[36228]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T02:07:47.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:47 smithi178 ceph-mon[36521]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-28T02:07:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:47 smithi153 ceph-mon[36228]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-28T02:07:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:47 smithi106 ceph-mon[31965]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-28T02:07:49.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:49 smithi178 ceph-mon[36521]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:49.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:49.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:49.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:49 smithi153 ceph-mon[36228]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:49 smithi106 ceph-mon[31965]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:51.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:51 smithi178 ceph-mon[36521]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:51 smithi153 ceph-mon[36228]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:51 smithi106 ceph-mon[31965]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:52.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.282+0000 7f27557fa700 1 -- 172.21.15.178:0/2928483969 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f27580625d0 con 0x7f27400a2890 2024-03-28T02:07:52.276 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 11 on host 'smithi178' 2024-03-28T02:07:52.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.285+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27400a2890 msgr2=0x7f27400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:52.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.285+0000 7f275e870700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27400a2890 0x7f27400a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f274c006fd0 tx=0x7f274c008040 comp rx=0 tx=0).stop 2024-03-28T02:07:52.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103e00 msgr2=0x7f2758101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:52.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103e00 0x7f2758101950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f275000b970 tx=0x7f275000bd30 comp rx=0 tx=0).stop 2024-03-28T02:07:52.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 shutdown_connections 2024-03-28T02:07:52.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2758103e00 0x7f2758101950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:52.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2758100480 0x7f2758102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:52.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27400a2890 0x7f27400a4d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:52.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 --2- 172.21.15.178:0/2928483969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2758103460 0x7f2758101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:52.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 >> 172.21.15.178:0/2928483969 conn(0x7f27580fbe00 msgr2=0x7f2758106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:07:52.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 shutdown_connections 2024-03-28T02:07:52.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:07:52.286+0000 7f275e870700 1 -- 172.21.15.178:0/2928483969 wait complete. 2024-03-28T02:07:52.992 DEBUG:teuthology.orchestra.run.smithi178:osd.11> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T02:07:52.994 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T02:07:52.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T02:07:53.249 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:07:53.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:53 smithi106 ceph-mon[31965]: from='osd.11 [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T02:07:53.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:53 smithi106 ceph-mon[31965]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T02:07:53.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:53 smithi106 ceph-mon[31965]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:53.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:53 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:53 smithi153 ceph-mon[36228]: from='osd.11 [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T02:07:53.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:53 smithi153 ceph-mon[36228]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T02:07:53.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:53 smithi153 ceph-mon[36228]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:53.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:53 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:53 smithi178 ceph-mon[36521]: from='osd.11 [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T02:07:53.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:53 smithi178 ceph-mon[36521]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T02:07:53.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:53 smithi178 ceph-mon[36521]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:53.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:53.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:53 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:54.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:54 smithi153 ceph-mon[36228]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T02:07:54.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:54 smithi153 ceph-mon[36228]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T02:07:54.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:54 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:54.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:54 smithi153 ceph-mon[36228]: from='osd.11 [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:07:54.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:54 smithi153 ceph-mon[36228]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:07:54.569 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:54 smithi106 ceph-mon[31965]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T02:07:54.569 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:54 smithi106 ceph-mon[31965]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T02:07:54.569 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:54 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:54.569 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:54 smithi106 ceph-mon[31965]: from='osd.11 [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:07:54.569 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:54 smithi106 ceph-mon[31965]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:07:54.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:54 smithi178 ceph-mon[36521]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T02:07:54.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:54 smithi178 ceph-mon[36521]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T02:07:54.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:54 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:54.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:54 smithi178 ceph-mon[36521]: from='osd.11 [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:07:54.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:54 smithi178 ceph-mon[36521]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-28T02:07:54.971 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:07:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:07:54.700+0000 7f3055719700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T02:07:54.971 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:07:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:07:54.712+0000 7f304d504700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T02:07:55.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.129+0000 7fa960661700 1 -- 172.21.15.106:0/3598199221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 msgr2=0x7fa9581025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.129+0000 7fa960661700 1 --2- 172.21.15.106:0/3598199221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 0x7fa9581025e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9540099c0 tx=0x7fa954009cd0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 -- 172.21.15.106:0/3598199221 shutdown_connections 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 --2- 172.21.15.106:0/3598199221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa958103d60 0x7fa95810a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 --2- 172.21.15.106:0/3598199221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 0x7fa9581025e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 --2- 172.21.15.106:0/3598199221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9581033c0 0x7fa958103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 -- 172.21.15.106:0/3598199221 >> 172.21.15.106:0/3598199221 conn(0x7fa9580fd7a0 msgr2=0x7fa9580ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 -- 172.21.15.106:0/3598199221 shutdown_connections 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 -- 172.21.15.106:0/3598199221 wait complete. 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 Processor -- start 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 -- start start 2024-03-28T02:07:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.130+0000 7fa960661700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 0x7fa958198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa960661700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9581033c0 0x7fa958198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa960661700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa958103d60 0x7fa95819f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa960661700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95810dda0 con 0x7fa9581021c0 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa960661700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95810dc20 con 0x7fa958103d60 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa960661700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95810df20 con 0x7fa9581033c0 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95e3fd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 0x7fa958198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95e3fd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 0x7fa958198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50626/0 (socket says 172.21.15.106:50626) 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95e3fd700 1 -- 172.21.15.106:0/932407176 learned_addr learned my addr 172.21.15.106:0/932407176 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95ebfe700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa958103d60 0x7fa95819f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95dbfc700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9581033c0 0x7fa958198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95e3fd700 1 -- 172.21.15.106:0/932407176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9581033c0 msgr2=0x7fa958198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95e3fd700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9581033c0 0x7fa958198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95e3fd700 1 -- 172.21.15.106:0/932407176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa958103d60 msgr2=0x7fa95819f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95e3fd700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa958103d60 0x7fa95819f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95e3fd700 1 -- 172.21.15.106:0/932407176 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9581a0090 con 0x7fa9581021c0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa95e3fd700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 0x7fa958198430 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa954005b30 tx=0x7fa954009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa94f7fe700 1 -- 172.21.15.106:0/932407176 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954027070 con 0x7fa9581021c0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa94f7fe700 1 -- 172.21.15.106:0/932407176 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa954004150 con 0x7fa9581021c0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.131+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9581a0320 con 0x7fa9581021c0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.132+0000 7fa94f7fe700 1 -- 172.21.15.106:0/932407176 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954022ce0 con 0x7fa9581021c0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.132+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9581a0800 con 0x7fa9581021c0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.133+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa958068e50 con 0x7fa9581021c0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.133+0000 7fa94f7fe700 1 -- 172.21.15.106:0/932407176 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa95402a070 con 0x7fa9581021c0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.134+0000 7fa94f7fe700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa9440a2890 0x7fa9440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.134+0000 7fa95dbfc700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa9440a2890 0x7fa9440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.134+0000 7fa94f7fe700 1 -- 172.21.15.106:0/932407176 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6847+0+0 (secure 0 0 0) 0x7fa9540cf3b0 con 0x7fa9581021c0 2024-03-28T02:07:55.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.135+0000 7fa95dbfc700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa9440a2890 0x7fa9440a4d50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa948007c60 tx=0x7fa9480073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:07:55.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.139+0000 7fa94f7fe700 1 -- 172.21.15.106:0/932407176 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa95405e740 con 0x7fa9581021c0 2024-03-28T02:07:55.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.345+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa958062da0 con 0x7fa9581021c0 2024-03-28T02:07:55.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.346+0000 7fa94f7fe700 1 -- 172.21.15.106:0/932407176 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7fa954013070 con 0x7fa9581021c0 2024-03-28T02:07:55.339 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:07:55.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.348+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa9440a2890 msgr2=0x7fa9440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:55.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.348+0000 7fa960661700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa9440a2890 0x7fa9440a4d50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa948007c60 tx=0x7fa9480073d0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.348+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 msgr2=0x7fa958198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:55.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.348+0000 7fa960661700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 0x7fa958198430 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa954005b30 tx=0x7fa954009580 comp rx=0 tx=0).stop 2024-03-28T02:07:55.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.349+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 shutdown_connections 2024-03-28T02:07:55.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.349+0000 7fa960661700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa958103d60 0x7fa95819f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.349+0000 7fa960661700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa9440a2890 0x7fa9440a4d50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.349+0000 7fa960661700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9581021c0 0x7fa958198430 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.349+0000 7fa960661700 1 --2- 172.21.15.106:0/932407176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9581033c0 0x7fa958198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:55.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.349+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 >> 172.21.15.106:0/932407176 conn(0x7fa9580fd7a0 msgr2=0x7fa958106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:07:55.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.349+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 shutdown_connections 2024-03-28T02:07:55.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:55.349+0000 7fa960661700 1 -- 172.21.15.106:0/932407176 wait complete. 2024-03-28T02:07:55.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:55 smithi106 ceph-mon[31965]: purged_snaps scrub starts 2024-03-28T02:07:55.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:55 smithi106 ceph-mon[31965]: purged_snaps scrub ok 2024-03-28T02:07:55.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:55 smithi106 ceph-mon[31965]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:07:55.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:55 smithi106 ceph-mon[31965]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T02:07:55.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:55.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:55 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:55.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:55 smithi106 ceph-mon[31965]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:55.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:55 smithi106 ceph-mon[31965]: from='osd.11 ' entity='osd.11' 2024-03-28T02:07:55.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:55 smithi178 ceph-mon[36521]: purged_snaps scrub starts 2024-03-28T02:07:55.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:55 smithi178 ceph-mon[36521]: purged_snaps scrub ok 2024-03-28T02:07:55.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:55 smithi178 ceph-mon[36521]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:07:55.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:55 smithi178 ceph-mon[36521]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T02:07:55.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:55 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:55.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:55 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:55.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:55 smithi178 ceph-mon[36521]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:55.624 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:55 smithi178 ceph-mon[36521]: from='osd.11 ' entity='osd.11' 2024-03-28T02:07:55.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:55 smithi153 ceph-mon[36228]: purged_snaps scrub starts 2024-03-28T02:07:55.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:55 smithi153 ceph-mon[36228]: purged_snaps scrub ok 2024-03-28T02:07:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:55 smithi153 ceph-mon[36228]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-28T02:07:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:55 smithi153 ceph-mon[36228]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T02:07:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:55 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:55 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:55 smithi153 ceph-mon[36228]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T02:07:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:55 smithi153 ceph-mon[36228]: from='osd.11 ' entity='osd.11' 2024-03-28T02:07:56.047 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1711591648,"num_in_osds":12,"osd_in_since":1711591654,"num_remapped_pgs":0} 2024-03-28T02:07:56.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:56.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:56 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/932407176' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T02:07:56.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:56 smithi106 ceph-mon[31965]: osd.11 [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] boot 2024-03-28T02:07:56.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:56 smithi106 ceph-mon[31965]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T02:07:56.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:56 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:56.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:56.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:56 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/932407176' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T02:07:56.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:56 smithi178 ceph-mon[36521]: osd.11 [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] boot 2024-03-28T02:07:56.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:56 smithi178 ceph-mon[36521]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T02:07:56.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:56 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:56.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:56.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:56 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/932407176' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T02:07:56.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:56 smithi153 ceph-mon[36228]: osd.11 [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] boot 2024-03-28T02:07:56.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:56 smithi153 ceph-mon[36228]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T02:07:56.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:56 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:07:57.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T02:07:57.289 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:07:57.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:07:57.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T02:07:57.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:57.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:57.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:57 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:57.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:07:57.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T02:07:57.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:57.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:57.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:57 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:57.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:07:57.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:57.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:57 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:07:58.541 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:58 smithi106 ceph-mon[31965]: Detected new or changed devices on smithi178 2024-03-28T02:07:58.541 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:58 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi178 to 4270M 2024-03-28T02:07:58.541 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:58.541 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:58.541 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:58 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:58.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:58 smithi178 ceph-mon[36521]: Detected new or changed devices on smithi178 2024-03-28T02:07:58.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:58 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi178 to 4270M 2024-03-28T02:07:58.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:58 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:58.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:58 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:58.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:58 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:58.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:58 smithi153 ceph-mon[36228]: Detected new or changed devices on smithi178 2024-03-28T02:07:58.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:58 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi178 to 4270M 2024-03-28T02:07:58.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:58 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:07:58.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:58 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:07:58.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:58 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.084+0000 7f08d354b700 1 -- 172.21.15.106:0/2452437537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 msgr2=0x7f08cc108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.084+0000 7f08d354b700 1 --2- 172.21.15.106:0/2452437537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 0x7f08cc108cb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f08c400b0d0 tx=0x7f08c400b3e0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.084+0000 7f08d354b700 1 -- 172.21.15.106:0/2452437537 shutdown_connections 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.084+0000 7f08d354b700 1 --2- 172.21.15.106:0/2452437537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 0x7f08cc108cb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.084+0000 7f08d354b700 1 --2- 172.21.15.106:0/2452437537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08cc100860 0x7f08cc100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.084+0000 7f08d354b700 1 --2- 172.21.15.106:0/2452437537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08cc101a60 0x7f08cc101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.084+0000 7f08d354b700 1 -- 172.21.15.106:0/2452437537 >> 172.21.15.106:0/2452437537 conn(0x7f08cc0fbe20 msgr2=0x7f08cc0fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.084+0000 7f08d354b700 1 -- 172.21.15.106:0/2452437537 shutdown_connections 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.084+0000 7f08d354b700 1 -- 172.21.15.106:0/2452437537 wait complete. 2024-03-28T02:07:59.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d354b700 1 Processor -- start 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d354b700 1 -- start start 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d354b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08cc100860 0x7f08cc1047f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d354b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08cc101a60 0x7f08cc102e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d354b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 0x7f08cc103380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d354b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc10cd50 con 0x7f08cc102400 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d354b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc10cbd0 con 0x7f08cc100860 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d354b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc10ced0 con 0x7f08cc101a60 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d1ae8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 0x7f08cc103380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d1ae8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 0x7f08cc103380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49054/0 (socket says 172.21.15.106:49054) 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d1ae8700 1 -- 172.21.15.106:0/936081515 learned_addr learned my addr 172.21.15.106:0/936081515 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d0ae6700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08cc101a60 0x7f08cc102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.085+0000 7f08d12e7700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08cc100860 0x7f08cc1047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:59.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08d1ae8700 1 -- 172.21.15.106:0/936081515 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08cc101a60 msgr2=0x7f08cc102e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08d1ae8700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08cc101a60 0x7f08cc102e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08d1ae8700 1 -- 172.21.15.106:0/936081515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08cc100860 msgr2=0x7f08cc1047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08d1ae8700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08cc100860 0x7f08cc1047f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08d1ae8700 1 -- 172.21.15.106:0/936081515 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc103c30 con 0x7f08cc102400 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08d0ae6700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08cc101a60 0x7f08cc102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08d1ae8700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 0x7f08cc103380 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f08c4006070 tx=0x7f08c4004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08c27fc700 1 -- 172.21.15.106:0/936081515 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4021070 con 0x7f08cc102400 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc1a2580 con 0x7f08cc102400 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08c27fc700 1 -- 172.21.15.106:0/936081515 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08c4007c80 con 0x7f08cc102400 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08c27fc700 1 -- 172.21.15.106:0/936081515 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4005480 con 0x7f08cc102400 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.086+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc1a2a00 con 0x7f08cc102400 2024-03-28T02:07:59.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.087+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc068e50 con 0x7f08cc102400 2024-03-28T02:07:59.084 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.088+0000 7f08c27fc700 1 -- 172.21.15.106:0/936081515 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f08c4024070 con 0x7f08cc102400 2024-03-28T02:07:59.084 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.088+0000 7f08c27fc700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f08b80a2890 0x7f08b80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:07:59.084 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.089+0000 7f08d12e7700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f08b80a2890 0x7f08b80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:07:59.084 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.089+0000 7f08c27fc700 1 -- 172.21.15.106:0/936081515 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f08c40cf800 con 0x7f08cc102400 2024-03-28T02:07:59.084 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.089+0000 7f08d12e7700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f08b80a2890 0x7f08b80a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f08bc006fd0 tx=0x7f08bc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:07:59.086 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.093+0000 7f08c27fc700 1 -- 172.21.15.106:0/936081515 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f08c409cdd0 con 0x7f08cc102400 2024-03-28T02:07:59.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.294+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f08cc1a2e50 con 0x7f08cc102400 2024-03-28T02:07:59.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.295+0000 7f08c27fc700 1 -- 172.21.15.106:0/936081515 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f08c409c6d0 con 0x7f08cc102400 2024-03-28T02:07:59.288 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:07:59.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f08b80a2890 msgr2=0x7f08b80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:59.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f08b80a2890 0x7f08b80a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f08bc006fd0 tx=0x7f08bc008040 comp rx=0 tx=0).stop 2024-03-28T02:07:59.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 msgr2=0x7f08cc103380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:07:59.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 0x7f08cc103380 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f08c4006070 tx=0x7f08c4004e70 comp rx=0 tx=0).stop 2024-03-28T02:07:59.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 shutdown_connections 2024-03-28T02:07:59.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f08b80a2890 0x7f08b80a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08cc102400 0x7f08cc103380 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08cc100860 0x7f08cc1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 --2- 172.21.15.106:0/936081515 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08cc101a60 0x7f08cc102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:07:59.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 >> 172.21.15.106:0/936081515 conn(0x7f08cc0fbe20 msgr2=0x7f08cc0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:07:59.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 shutdown_connections 2024-03-28T02:07:59.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:07:59.297+0000 7f08d354b700 1 -- 172.21.15.106:0/936081515 wait complete. 2024-03-28T02:07:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:07:59 smithi106 ceph-mon[31965]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:07:59.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:07:59 smithi178 ceph-mon[36521]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:07:59.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:07:59 smithi153 ceph-mon[36228]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:07:59.951 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1711591675,"num_in_osds":12,"osd_in_since":1711591654,"num_remapped_pgs":0} 2024-03-28T02:07:59.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T02:08:00.190 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:08:00.553 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:00 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/936081515' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T02:08:00.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:00 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/936081515' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T02:08:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:00 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/936081515' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T02:08:01.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:01 smithi106 ceph-mon[31965]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:01.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:01 smithi178 ceph-mon[36521]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:01.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:01 smithi153 ceph-mon[36228]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:01.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.986+0000 7fb1b4552700 1 -- 172.21.15.106:0/3722924986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac107280 msgr2=0x7fb1ac107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:01.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.986+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3722924986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac107280 0x7fb1ac107660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009990 tx=0x7fb1a0009ca0 comp rx=0 tx=0).stop 2024-03-28T02:08:01.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.986+0000 7fb1b4552700 1 -- 172.21.15.106:0/3722924986 shutdown_connections 2024-03-28T02:08:01.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.986+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3722924986 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1ac101d30 0x7fb1ac10df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:01.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.986+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3722924986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac107280 0x7fb1ac107660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:01.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.986+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3722924986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ac101370 0x7fb1ac1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:01.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.986+0000 7fb1b4552700 1 -- 172.21.15.106:0/3722924986 >> 172.21.15.106:0/3722924986 conn(0x7fb1ac077ef0 msgr2=0x7fb1ac078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:01.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.987+0000 7fb1b4552700 1 -- 172.21.15.106:0/3722924986 shutdown_connections 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.987+0000 7fb1b4552700 1 -- 172.21.15.106:0/3722924986 wait complete. 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.987+0000 7fb1b4552700 1 Processor -- start 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.987+0000 7fb1b4552700 1 -- start start 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b4552700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ac101370 0x7fb1ac19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b4552700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac101d30 0x7fb1ac19dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b4552700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1ac107280 0x7fb1ac197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b4552700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac111460 con 0x7fb1ac101d30 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b4552700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac1112e0 con 0x7fb1ac101370 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b4552700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac1115e0 con 0x7fb1ac107280 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b1aed700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac101d30 0x7fb1ac19dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b1aed700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac101d30 0x7fb1ac19dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49072/0 (socket says 172.21.15.106:49072) 2024-03-28T02:08:01.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b1aed700 1 -- 172.21.15.106:0/3241899275 learned_addr learned my addr 172.21.15.106:0/3241899275 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b22ee700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ac101370 0x7fb1ac19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b2aef700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1ac107280 0x7fb1ac197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b1aed700 1 -- 172.21.15.106:0/3241899275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1ac107280 msgr2=0x7fb1ac197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b1aed700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1ac107280 0x7fb1ac197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b1aed700 1 -- 172.21.15.106:0/3241899275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ac101370 msgr2=0x7fb1ac19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b1aed700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ac101370 0x7fb1ac19d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b1aed700 1 -- 172.21.15.106:0/3241899275 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac198020 con 0x7fb1ac101d30 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b1aed700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac101d30 0x7fb1ac19dbb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800d950 tx=0x7fb1a800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb19f7fe700 1 -- 172.21.15.106:0/3241899275 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8016070 con 0x7fb1ac101d30 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb19f7fe700 1 -- 172.21.15.106:0/3241899275 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1a800fd70 con 0x7fb1ac101d30 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac198310 con 0x7fb1ac101d30 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb19f7fe700 1 -- 172.21.15.106:0/3241899275 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a800e610 con 0x7fb1ac101d30 2024-03-28T02:08:01.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.988+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ac1a45f0 con 0x7fb1ac101d30 2024-03-28T02:08:01.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.989+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ac102810 con 0x7fb1ac101d30 2024-03-28T02:08:01.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.990+0000 7fb19f7fe700 1 -- 172.21.15.106:0/3241899275 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb1a8019070 con 0x7fb1ac101d30 2024-03-28T02:08:01.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.991+0000 7fb19f7fe700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1980a2910 0x7fb1980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:01.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.991+0000 7fb1b22ee700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1980a2910 0x7fb1980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:01.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.991+0000 7fb19f7fe700 1 -- 172.21.15.106:0/3241899275 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb1a80cdfc0 con 0x7fb1ac101d30 2024-03-28T02:08:01.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.991+0000 7fb1b22ee700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1980a2910 0x7fb1980a4dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000b5c0 tx=0x7fb1a0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:01.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:01.995+0000 7fb19f7fe700 1 -- 172.21.15.106:0/3241899275 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb1a809b500 con 0x7fb1ac101d30 2024-03-28T02:08:02.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.190+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1ac198c70 con 0x7fb1ac101d30 2024-03-28T02:08:02.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.191+0000 7fb19f7fe700 1 -- 172.21.15.106:0/3241899275 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18534 (secure 0 0 0) 0x7fb1a809ae00 con 0x7fb1ac101d30 2024-03-28T02:08:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:02.185 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":81,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:07:56.706770+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1980a2910 msgr2=0x7fb1980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1980a2910 0x7fb1980a4dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000b5c0 tx=0x7fb1a0005960 comp rx=0 tx=0).stop 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac101d30 msgr2=0x7fb1ac19dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac101d30 0x7fb1ac19dbb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800d950 tx=0x7fb1a800dc60 comp rx=0 tx=0).stop 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 shutdown_connections 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1ac107280 0x7fb1ac197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ac101370 0x7fb1ac19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1980a2910 0x7fb1980a4dd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 --2- 172.21.15.106:0/3241899275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1ac101d30 0x7fb1ac19dbb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 >> 172.21.15.106:0/3241899275 conn(0x7fb1ac077ef0 msgr2=0x7fb1ac0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:02.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 shutdown_connections 2024-03-28T02:08:02.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:02.193+0000 7fb1b4552700 1 -- 172.21.15.106:0/3241899275 wait complete. 2024-03-28T02:08:02.500 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3241899275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:02.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3241899275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:02.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3241899275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:02.963 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T02:08:02.963 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T02:08:03.200 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:08:03.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:03 smithi106 ceph-mon[31965]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:03.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:03 smithi178 ceph-mon[36521]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:03.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:03 smithi153 ceph-mon[36228]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:04.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 -- 172.21.15.106:0/1254604027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 msgr2=0x7fcda0106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:04.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/1254604027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 0x7fcda0106e00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0099c0 tx=0x7fcd8c009cd0 comp rx=0 tx=0).stop 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 -- 172.21.15.106:0/1254604027 shutdown_connections 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/1254604027 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcda01074e0 0x7fcda010f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/1254604027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcda006aba0 0x7fcda006af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/1254604027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 0x7fcda0106e00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 -- 172.21.15.106:0/1254604027 >> 172.21.15.106:0/1254604027 conn(0x7fcda0078160 msgr2=0x7fcda0078570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 -- 172.21.15.106:0/1254604027 shutdown_connections 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 -- 172.21.15.106:0/1254604027 wait complete. 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 Processor -- start 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 -- start start 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcda006aba0 0x7fcda01a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.943+0000 7fcda6ad2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 0x7fcda01a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcda6ad2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcda01074e0 0x7fcda019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcda6ad2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda0112540 con 0x7fcda006b550 2024-03-28T02:08:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcda6ad2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda01123c0 con 0x7fcda006aba0 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcda6ad2700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda01126c0 con 0x7fcda01074e0 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 0x7fcda01a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 0x7fcda01a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49084/0 (socket says 172.21.15.106:49084) 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9ffff700 1 -- 172.21.15.106:0/3729884081 learned_addr learned my addr 172.21.15.106:0/3729884081 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9ffff700 1 -- 172.21.15.106:0/3729884081 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcda01074e0 msgr2=0x7fcda019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9ffff700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcda01074e0 0x7fcda019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9ffff700 1 -- 172.21.15.106:0/3729884081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcda006aba0 msgr2=0x7fcda01a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9ffff700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcda006aba0 0x7fcda01a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9ffff700 1 -- 172.21.15.106:0/3729884081 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda019b160 con 0x7fcda006b550 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9ffff700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 0x7fcda01a0d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009fd0 tx=0x7fcd8c005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9dffb700 1 -- 172.21.15.106:0/3729884081 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c027070 con 0x7fcda006b550 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda019b3f0 con 0x7fcda006b550 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9dffb700 1 -- 172.21.15.106:0/3729884081 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd8c004050 con 0x7fcda006b550 2024-03-28T02:08:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcd9dffb700 1 -- 172.21.15.106:0/3729884081 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c022e00 con 0x7fcda006b550 2024-03-28T02:08:04.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.944+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda01a7710 con 0x7fcda006b550 2024-03-28T02:08:04.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.945+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda0068e50 con 0x7fcda006b550 2024-03-28T02:08:04.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.946+0000 7fcd9dffb700 1 -- 172.21.15.106:0/3729884081 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcd8c02a070 con 0x7fcda006b550 2024-03-28T02:08:04.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.947+0000 7fcd9dffb700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd900a2890 0x7fcd900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:04.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.947+0000 7fcda486e700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd900a2890 0x7fcd900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:04.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.947+0000 7fcd9dffb700 1 -- 172.21.15.106:0/3729884081 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcd8c0cf8f0 con 0x7fcda006b550 2024-03-28T02:08:04.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.947+0000 7fcda486e700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd900a2890 0x7fcd900a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcda019bb40 tx=0x7fcd94006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:04.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:04.951+0000 7fcd9dffb700 1 -- 172.21.15.106:0/3729884081 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcd8c013070 con 0x7fcda006b550 2024-03-28T02:08:05.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.142+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcda01a78e0 con 0x7fcda006b550 2024-03-28T02:08:05.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.142+0000 7fcd9dffb700 1 -- 172.21.15.106:0/3729884081 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcd8c09cec0 con 0x7fcda006b550 2024-03-28T02:08:05.136 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:08:05.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd900a2890 msgr2=0x7fcd900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:05.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd900a2890 0x7fcd900a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcda019bb40 tx=0x7fcd94006d20 comp rx=0 tx=0).stop 2024-03-28T02:08:05.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 msgr2=0x7fcda01a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:05.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 0x7fcda01a0d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009fd0 tx=0x7fcd8c005f10 comp rx=0 tx=0).stop 2024-03-28T02:08:05.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 shutdown_connections 2024-03-28T02:08:05.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcda01074e0 0x7fcda019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:05.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcda006aba0 0x7fcda01a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:05.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd900a2890 0x7fcd900a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:05.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 --2- 172.21.15.106:0/3729884081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcda006b550 0x7fcda01a0d80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:05.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 >> 172.21.15.106:0/3729884081 conn(0x7fcda0078160 msgr2=0x7fcda00fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:05.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 shutdown_connections 2024-03-28T02:08:05.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:05.145+0000 7fcda6ad2700 1 -- 172.21.15.106:0/3729884081 wait complete. 2024-03-28T02:08:05.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:05 smithi153 ceph-mon[36228]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:05.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:05 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3729884081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:08:05.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:05 smithi106 ceph-mon[31965]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:05.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:05 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3729884081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:08:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:05 smithi178 ceph-mon[36521]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:05 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3729884081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.a on smithi106 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.d on smithi106 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.g on smithi106 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.j on smithi106 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.b on smithi153 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.e on smithi153 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.h on smithi153 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.k on smithi153 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.c on smithi178 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.f on smithi178 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.i on smithi178 2024-03-28T02:08:05.912 INFO:tasks.cephadm:Adding mds.l on smithi178 2024-03-28T02:08:05.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi106=a;smithi106=d;smithi106=g;smithi106=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l' 2024-03-28T02:08:06.161 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:08:07.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:07 smithi178 ceph-mon[36521]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:07.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:07 smithi153 ceph-mon[36228]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:07 smithi106 ceph-mon[31965]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:07.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 -- 172.21.15.178:0/4227163416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c102650 msgr2=0x7f599c102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:07.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 --2- 172.21.15.178:0/4227163416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c102650 0x7f599c102a70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f598c0099c0 tx=0x7f598c009cd0 comp rx=0 tx=0).stop 2024-03-28T02:08:07.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 -- 172.21.15.178:0/4227163416 shutdown_connections 2024-03-28T02:08:07.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 --2- 172.21.15.178:0/4227163416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599c101d30 0x7f599c1095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:07.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 --2- 172.21.15.178:0/4227163416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c102650 0x7f599c102a70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 --2- 172.21.15.178:0/4227163416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f599c101390 0x7f599c1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 -- 172.21.15.178:0/4227163416 >> 172.21.15.178:0/4227163416 conn(0x7f599c0fbe00 msgr2=0x7f599c0fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 -- 172.21.15.178:0/4227163416 shutdown_connections 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 -- 172.21.15.178:0/4227163416 wait complete. 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.879+0000 7f59a153b700 1 Processor -- start 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59a153b700 1 -- start start 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59a153b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f599c101390 0x7f599c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59a153b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c101d30 0x7f599c198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59a153b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599c102650 0x7f599c19faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59a153b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c10cd80 con 0x7f599c101390 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59a153b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c10cc00 con 0x7f599c102650 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59a153b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c10cf00 con 0x7f599c101d30 2024-03-28T02:08:07.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59927fc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c101d30 0x7f599c198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59927fc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c101d30 0x7f599c198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:34650/0 (socket says 172.21.15.178:34650) 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59927fc700 1 -- 172.21.15.178:0/4022010738 learned_addr learned my addr 172.21.15.178:0/4022010738 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f599affd700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f599c101390 0x7f599c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f599b7fe700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599c102650 0x7f599c19faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59927fc700 1 -- 172.21.15.178:0/4022010738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599c102650 msgr2=0x7f599c19faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59927fc700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599c102650 0x7f599c19faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59927fc700 1 -- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f599c101390 msgr2=0x7f599c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59927fc700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f599c101390 0x7f599c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59927fc700 1 -- 172.21.15.178:0/4022010738 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c1a0190 con 0x7f599c101d30 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.880+0000 7f59927fc700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c101d30 0x7f599c198ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f598400b560 tx=0x7f598400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.881+0000 7f5998ff9700 1 -- 172.21.15.178:0/4022010738 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984015020 con 0x7f599c101d30 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.881+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c1a0480 con 0x7f599c101d30 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.881+0000 7f5998ff9700 1 -- 172.21.15.178:0/4022010738 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5984011d10 con 0x7f599c101d30 2024-03-28T02:08:07.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.881+0000 7f5998ff9700 1 -- 172.21.15.178:0/4022010738 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984010e50 con 0x7f599c101d30 2024-03-28T02:08:07.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.881+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599c1a0a90 con 0x7f599c101d30 2024-03-28T02:08:07.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.882+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599c068e50 con 0x7f599c101d30 2024-03-28T02:08:07.878 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.883+0000 7f5998ff9700 1 -- 172.21.15.178:0/4022010738 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5984018020 con 0x7f599c101d30 2024-03-28T02:08:07.878 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.883+0000 7f5998ff9700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f59880a2a30 0x7f59880a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:07.878 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.884+0000 7f5998ff9700 1 -- 172.21.15.178:0/4022010738 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f59840ce220 con 0x7f599c101d30 2024-03-28T02:08:07.878 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.884+0000 7f599affd700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f59880a2a30 0x7f59880a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:07.878 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.884+0000 7f599affd700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f59880a2a30 0x7f59880a4ef0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f598c00b580 tx=0x7f598c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:07.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:07.887+0000 7f5998ff9700 1 -- 172.21.15.178:0/4022010738 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f598401f020 con 0x7f599c101d30 2024-03-28T02:08:08.082 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.089+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi106=a;smithi106=d;smithi106=g;smithi106=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f599c100a50 con 0x7f59880a2a30 2024-03-28T02:08:08.090 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.097+0000 7f5998ff9700 1 -- 172.21.15.178:0/4022010738 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f599c100a50 con 0x7f59880a2a30 2024-03-28T02:08:08.091 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mds.all update... 2024-03-28T02:08:08.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f59880a2a30 msgr2=0x7f59880a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:08.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f59880a2a30 0x7f59880a4ef0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f598c00b580 tx=0x7f598c005fb0 comp rx=0 tx=0).stop 2024-03-28T02:08:08.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c101d30 msgr2=0x7f599c198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:08.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c101d30 0x7f599c198ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f598400b560 tx=0x7f598400b870 comp rx=0 tx=0).stop 2024-03-28T02:08:08.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 shutdown_connections 2024-03-28T02:08:08.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599c102650 0x7f599c19faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:08.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f59880a2a30 0x7f59880a4ef0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:08.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f599c101390 0x7f599c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:08.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 --2- 172.21.15.178:0/4022010738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f599c101d30 0x7f599c198ac0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:08.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.099+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 >> 172.21.15.178:0/4022010738 conn(0x7f599c0fbe00 msgr2=0x7f599c100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:08.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.100+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 shutdown_connections 2024-03-28T02:08:08.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-28T02:08:08.100+0000 7f59a153b700 1 -- 172.21.15.178:0/4022010738 wait complete. 2024-03-28T02:08:08.759 DEBUG:teuthology.orchestra.run.smithi106:mds.a> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T02:08:08.762 DEBUG:teuthology.orchestra.run.smithi106:mds.d> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T02:08:08.805 DEBUG:teuthology.orchestra.run.smithi106:mds.g> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T02:08:08.848 DEBUG:teuthology.orchestra.run.smithi106:mds.j> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T02:08:08.891 DEBUG:teuthology.orchestra.run.smithi153:mds.b> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T02:08:08.894 DEBUG:teuthology.orchestra.run.smithi153:mds.e> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T02:08:08.937 DEBUG:teuthology.orchestra.run.smithi153:mds.h> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T02:08:08.980 DEBUG:teuthology.orchestra.run.smithi153:mds.k> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T02:08:09.023 DEBUG:teuthology.orchestra.run.smithi178:mds.c> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T02:08:09.025 DEBUG:teuthology.orchestra.run.smithi178:mds.f> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T02:08:09.067 DEBUG:teuthology.orchestra.run.smithi178:mds.i> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T02:08:09.111 DEBUG:teuthology.orchestra.run.smithi178:mds.l> sudo journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T02:08:09.154 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T02:08:09.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi106=a;smithi106=d;smithi106=g;smithi106=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: Saving service mds.all spec with placement smithi106=a;smithi106=d;smithi106=g;smithi106=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:09.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: Deploying daemon mds.l on smithi178 2024-03-28T02:08:09.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:09 smithi153 ceph-mon[36228]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi106=a;smithi106=d;smithi106=g;smithi106=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: Saving service mds.all spec with placement smithi106=a;smithi106=d;smithi106=g;smithi106=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:09.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: Deploying daemon mds.l on smithi178 2024-03-28T02:08:09.236 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:09 smithi178 ceph-mon[36521]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:09.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 -- 172.21.15.106:0/2068136016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 msgr2=0x7fb2b8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 --2- 172.21.15.106:0/2068136016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 0x7fb2b8101c30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009900 tx=0x7fb2ac02bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:09.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 -- 172.21.15.106:0/2068136016 shutdown_connections 2024-03-28T02:08:09.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 --2- 172.21.15.106:0/2068136016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b8102170 0x7fb2b8108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 --2- 172.21.15.106:0/2068136016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 0x7fb2b8101c30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 --2- 172.21.15.106:0/2068136016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2b81005c0 0x7fb2b81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 -- 172.21.15.106:0/2068136016 >> 172.21.15.106:0/2068136016 conn(0x7fb2b80fbb50 msgr2=0x7fb2b80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 -- 172.21.15.106:0/2068136016 shutdown_connections 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 -- 172.21.15.106:0/2068136016 wait complete. 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 Processor -- start 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.282+0000 7fb2beef1640 1 -- start start 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2beef1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2b81005c0 0x7fb2b81044e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2beef1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 0x7fb2b8102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2beef1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b8102170 0x7fb2b81030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2beef1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b810c2a0 con 0x7fb2b81017b0 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2beef1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b810c120 con 0x7fb2b8102170 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2beef1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b810c420 con 0x7fb2b81005c0 2024-03-28T02:08:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 0x7fb2b8102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2bcc8d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2b81005c0 0x7fb2b81044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 0x7fb2b8102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:45468/0 (socket says 172.21.15.106:45468) 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b7fff640 1 -- 172.21.15.106:0/957812996 learned_addr learned my addr 172.21.15.106:0/957812996 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2bd48e640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b8102170 0x7fb2b81030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b7fff640 1 -- 172.21.15.106:0/957812996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2b81005c0 msgr2=0x7fb2b81044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b7fff640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2b81005c0 0x7fb2b81044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b7fff640 1 -- 172.21.15.106:0/957812996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b8102170 msgr2=0x7fb2b81030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b7fff640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b8102170 0x7fb2b81030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b7fff640 1 -- 172.21.15.106:0/957812996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b81038f0 con 0x7fb2b81017b0 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2bd48e640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b8102170 0x7fb2b81030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b7fff640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 0x7fb2b8102b90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009230 tx=0x7fb2ac005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b5ffb640 1 -- 172.21.15.106:0/957812996 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac041070 con 0x7fb2b81017b0 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b81a1b30 con 0x7fb2b81017b0 2024-03-28T02:08:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b5ffb640 1 -- 172.21.15.106:0/957812996 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2ac0054e0 con 0x7fb2b81017b0 2024-03-28T02:08:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.283+0000 7fb2b5ffb640 1 -- 172.21.15.106:0/957812996 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac034440 con 0x7fb2b81017b0 2024-03-28T02:08:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.284+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b81a2070 con 0x7fb2b81017b0 2024-03-28T02:08:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.285+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b81a2240 con 0x7fb2b81017b0 2024-03-28T02:08:09.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.286+0000 7fb2b5ffb640 1 -- 172.21.15.106:0/957812996 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb2ac044070 con 0x7fb2b81017b0 2024-03-28T02:08:09.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.286+0000 7fb2b5ffb640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2a00a2800 0x7fb2a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.286+0000 7fb2bcc8d640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2a00a2800 0x7fb2a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.286+0000 7fb2b5ffb640 1 -- 172.21.15.106:0/957812996 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb2ac0f02c0 con 0x7fb2b81017b0 2024-03-28T02:08:09.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.287+0000 7fb2bcc8d640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2a00a2800 0x7fb2a00a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8005fd0 tx=0x7fb2a800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:09.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.291+0000 7fb2b5ffb640 1 -- 172.21.15.106:0/957812996 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb2ac0bd800 con 0x7fb2b81017b0 2024-03-28T02:08:09.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.468+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2b81a25d0 con 0x7fb2b81017b0 2024-03-28T02:08:09.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.469+0000 7fb2b5ffb640 1 -- 172.21.15.106:0/957812996 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18534 (secure 0 0 0) 0x7fb2ac0bd100 con 0x7fb2b81017b0 2024-03-28T02:08:09.463 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:09.463 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":81,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:07:56.706770+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:08:09.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2a00a2800 msgr2=0x7fb2a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2a00a2800 0x7fb2a00a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8005fd0 tx=0x7fb2a800b410 comp rx=0 tx=0).stop 2024-03-28T02:08:09.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 msgr2=0x7fb2b8102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 0x7fb2b8102b90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009230 tx=0x7fb2ac005dd0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 shutdown_connections 2024-03-28T02:08:09.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2a00a2800 0x7fb2a00a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b8102170 0x7fb2b81030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2b81017b0 0x7fb2b8102b90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 --2- 172.21.15.106:0/957812996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2b81005c0 0x7fb2b81044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 >> 172.21.15.106:0/957812996 conn(0x7fb2b80fbb50 msgr2=0x7fb2b80fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:09.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 shutdown_connections 2024-03-28T02:08:09.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.471+0000 7fb2beef1640 1 -- 172.21.15.106:0/957812996 wait complete. 2024-03-28T02:08:09.474 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi106=a;smithi106=d;smithi106=g;smithi106=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:08:09.474 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: Saving service mds.all spec with placement smithi106=a;smithi106=d;smithi106=g;smithi106=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-03-28T02:08:09.474 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:09.474 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:08:09.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:09.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:08:09.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:09.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:09.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:09.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:09.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: Deploying daemon mds.l on smithi178 2024-03-28T02:08:09.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:09 smithi106 ceph-mon[31965]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:09.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T02:08:09.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 -- 172.21.15.106:0/335659094 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44102150 msgr2=0x7f7e441089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/335659094 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44102150 0x7f7e441089e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009ca0 tx=0x7f7e3c02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 -- 172.21.15.106:0/335659094 shutdown_connections 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/335659094 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44102150 0x7f7e441089e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/335659094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e44101790 0x7f7e44101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/335659094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e441005a0 0x7f7e441009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 -- 172.21.15.106:0/335659094 >> 172.21.15.106:0/335659094 conn(0x7f7e440fbb30 msgr2=0x7f7e440fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 -- 172.21.15.106:0/335659094 shutdown_connections 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 -- 172.21.15.106:0/335659094 wait complete. 2024-03-28T02:08:09.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 Processor -- start 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.587+0000 7f7e4aad3640 1 -- start start 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e4aad3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e441005a0 0x7f7e44197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e4aad3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44101790 0x7f7e44197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e4aad3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e44102150 0x7f7e4419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e4aad3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4410d340 con 0x7f7e44101790 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e4aad3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4410d1c0 con 0x7f7e44102150 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e4aad3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4410d4c0 con 0x7f7e441005a0 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e43fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44101790 0x7f7e44197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e43fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44101790 0x7f7e44197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:45488/0 (socket says 172.21.15.106:45488) 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e43fff640 1 -- 172.21.15.106:0/3441872928 learned_addr learned my addr 172.21.15.106:0/3441872928 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e49070640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e44102150 0x7f7e4419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e4886f640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e441005a0 0x7f7e44197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e43fff640 1 -- 172.21.15.106:0/3441872928 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e441005a0 msgr2=0x7f7e44197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e43fff640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e441005a0 0x7f7e44197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e43fff640 1 -- 172.21.15.106:0/3441872928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e44102150 msgr2=0x7f7e4419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e43fff640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e44102150 0x7f7e4419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e43fff640 1 -- 172.21.15.106:0/3441872928 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4419f560 con 0x7f7e44101790 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e49070640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e44102150 0x7f7e4419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e43fff640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44101790 0x7f7e44197e80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000d950 tx=0x7f7e3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e41ffb640 1 -- 172.21.15.106:0/3441872928 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30017030 con 0x7f7e44101790 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4419f850 con 0x7f7e44101790 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e41ffb640 1 -- 172.21.15.106:0/3441872928 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e30004020 con 0x7f7e44101790 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.588+0000 7f7e41ffb640 1 -- 172.21.15.106:0/3441872928 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e300047a0 con 0x7f7e44101790 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.589+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4419fda0 con 0x7f7e44101790 2024-03-28T02:08:09.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.589+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e441078c0 con 0x7f7e44101790 2024-03-28T02:08:09.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.590+0000 7f7e41ffb640 1 -- 172.21.15.106:0/3441872928 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7e300041d0 con 0x7f7e44101790 2024-03-28T02:08:09.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.591+0000 7f7e41ffb640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7e340a2800 0x7f7e340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.591+0000 7f7e4886f640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7e340a2800 0x7f7e340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.591+0000 7f7e41ffb640 1 -- 172.21.15.106:0/3441872928 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7e300ce050 con 0x7f7e44101790 2024-03-28T02:08:09.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.591+0000 7f7e4886f640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7e340a2800 0x7f7e340a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7e441015f0 tx=0x7f7e38005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:09.589 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.595+0000 7f7e41ffb640 1 -- 172.21.15.106:0/3441872928 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7e3009b590 con 0x7f7e44101790 2024-03-28T02:08:09.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.769+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e4402d2d0 con 0x7f7e44101790 2024-03-28T02:08:09.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.769+0000 7f7e41ffb640 1 -- 172.21.15.106:0/3441872928 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e3009ae90 con 0x7f7e44101790 2024-03-28T02:08:09.763 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:08:09.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7e340a2800 msgr2=0x7f7e340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7e340a2800 0x7f7e340a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7e441015f0 tx=0x7f7e38005c60 comp rx=0 tx=0).stop 2024-03-28T02:08:09.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44101790 msgr2=0x7f7e44197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44101790 0x7f7e44197e80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000d950 tx=0x7f7e3000de10 comp rx=0 tx=0).stop 2024-03-28T02:08:09.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 shutdown_connections 2024-03-28T02:08:09.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7e340a2800 0x7f7e340a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e44102150 0x7f7e4419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e44101790 0x7f7e44197e80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 --2- 172.21.15.106:0/3441872928 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e441005a0 0x7f7e44197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.771+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 >> 172.21.15.106:0/3441872928 conn(0x7f7e440fbb30 msgr2=0x7f7e44104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:09.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.772+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 shutdown_connections 2024-03-28T02:08:09.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.772+0000 7f7e4aad3640 1 -- 172.21.15.106:0/3441872928 wait complete. 2024-03-28T02:08:09.776 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:09.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/1102265089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 msgr2=0x7fdb28102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/1102265089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 0x7fdb28102190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009900 tx=0x7fdb2402bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:09.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/1102265089 shutdown_connections 2024-03-28T02:08:09.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/1102265089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb28105080 0x7fdb28105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/1102265089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb281026d0 0x7fdb28104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/1102265089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 0x7fdb28102190 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/1102265089 >> 172.21.15.106:0/1102265089 conn(0x7fdb280fb980 msgr2=0x7fdb280fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/1102265089 shutdown_connections 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/1102265089 wait complete. 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.884+0000 7fdb2ed9d640 1 Processor -- start 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2ed9d640 1 -- start start 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2ed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 0x7fdb2819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2ed9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb281026d0 0x7fdb2819e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb28105080 0x7fdb281a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2ed9d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28113830 con 0x7fdb280ffd60 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2ed9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281136b0 con 0x7fdb28105080 2024-03-28T02:08:09.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2ed9d640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281139b0 con 0x7fdb281026d0 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 0x7fdb2819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 0x7fdb2819de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:45498/0 (socket says 172.21.15.106:45498) 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2dd9b640 1 -- 172.21.15.106:0/3366188571 learned_addr learned my addr 172.21.15.106:0/3366188571 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2dd9b640 1 -- 172.21.15.106:0/3366188571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb281026d0 msgr2=0x7fdb2819e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2dd9b640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb281026d0 0x7fdb2819e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2dd9b640 1 -- 172.21.15.106:0/3366188571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb28105080 msgr2=0x7fdb281a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2dd9b640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb28105080 0x7fdb281a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2dd9b640 1 -- 172.21.15.106:0/3366188571 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb281a5aa0 con 0x7fdb280ffd60 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2dd9b640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 0x7fdb2819de80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009230 tx=0x7fdb24004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb1effd640 1 -- 172.21.15.106:0/3366188571 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb24041070 con 0x7fdb280ffd60 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb281a5d30 con 0x7fdb280ffd60 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb1effd640 1 -- 172.21.15.106:0/3366188571 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb240054e0 con 0x7fdb280ffd60 2024-03-28T02:08:09.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb1effd640 1 -- 172.21.15.106:0/3366188571 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb240346a0 con 0x7fdb280ffd60 2024-03-28T02:08:09.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.885+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb281a61d0 con 0x7fdb280ffd60 2024-03-28T02:08:09.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.886+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb280fd470 con 0x7fdb280ffd60 2024-03-28T02:08:09.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.887+0000 7fdb1effd640 1 -- 172.21.15.106:0/3366188571 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdb24044070 con 0x7fdb280ffd60 2024-03-28T02:08:09.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.888+0000 7fdb1effd640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb140a2800 0x7fdb140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:09.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.888+0000 7fdb2d59a640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb140a2800 0x7fdb140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:09.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.888+0000 7fdb1effd640 1 -- 172.21.15.106:0/3366188571 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdb240f0830 con 0x7fdb280ffd60 2024-03-28T02:08:09.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.889+0000 7fdb2d59a640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb140a2800 0x7fdb140a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdb2819f390 tx=0x7fdb18006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:09.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:09.892+0000 7fdb1effd640 1 -- 172.21.15.106:0/3366188571 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdb240bdd50 con 0x7fdb280ffd60 2024-03-28T02:08:10.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.102+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb2802d070 con 0x7fdb280ffd60 2024-03-28T02:08:10.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.103+0000 7fdb1effd640 1 -- 172.21.15.106:0/3366188571 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fdb240bd650 con 0x7fdb280ffd60 2024-03-28T02:08:10.096 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:10.097 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T02:08:10.097 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 1 2024-03-28T02:08:10.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb140a2800 msgr2=0x7fdb140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb140a2800 0x7fdb140a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdb2819f390 tx=0x7fdb18006c90 comp rx=0 tx=0).stop 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 msgr2=0x7fdb2819de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 0x7fdb2819de80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009230 tx=0x7fdb24004040 comp rx=0 tx=0).stop 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 shutdown_connections 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb140a2800 0x7fdb140a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb28105080 0x7fdb281a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb281026d0 0x7fdb2819e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 --2- 172.21.15.106:0/3366188571 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb280ffd60 0x7fdb2819de80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 >> 172.21.15.106:0/3366188571 conn(0x7fdb280fb980 msgr2=0x7fdb28103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 shutdown_connections 2024-03-28T02:08:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:10.105+0000 7fdb2ed9d640 1 -- 172.21.15.106:0/3366188571 wait complete. 2024-03-28T02:08:10.301 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:10 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/957812996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:10.301 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:10 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3441872928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:08:10.301 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:10 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3366188571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:10.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:10 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/957812996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:10.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:10 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3441872928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:08:10.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:10 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3366188571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:10.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:10 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/957812996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:10.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:10 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3441872928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:08:10.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:10 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3366188571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:10.622 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:08:10 smithi178 systemd[1]: Starting Ceph mds.l for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:11.012 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:08:10 smithi178 podman[66715]: 2024-03-28 02:08:10.601120394 +0000 UTC m=+0.017433325 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:11.012 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:08:10 smithi178 podman[66715]: 2024-03-28 02:08:10.74170307 +0000 UTC m=+0.158016001 container create d8f3d10520669311aff7ebb6586234acfe4fe2e5698c9a3936249e09ddc782b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:08:11.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:11 smithi178 ceph-mon[36521]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:11.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:11 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] up:boot 2024-03-28T02:08:11.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:11 smithi178 ceph-mon[36521]: fsmap 1 up:standby 2024-03-28T02:08:11.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:11 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T02:08:11.372 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:08:11 smithi178 podman[66715]: 2024-03-28 02:08:11.019651885 +0000 UTC m=+0.435964817 container init d8f3d10520669311aff7ebb6586234acfe4fe2e5698c9a3936249e09ddc782b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-28T02:08:11.372 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:08:11 smithi178 podman[66715]: 2024-03-28 02:08:11.022378805 +0000 UTC m=+0.438691733 container start d8f3d10520669311aff7ebb6586234acfe4fe2e5698c9a3936249e09ddc782b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T02:08:11.373 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:08:11 smithi178 ceph-mds[66734]: -- 172.21.15.178:0/3113953642 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580e8c3fa40 con 0x5580e8c12800 2024-03-28T02:08:11.373 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:08:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l[66730]: starting mds.l at 2024-03-28T02:08:11.373 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:08:11 smithi178 bash[66715]: d8f3d10520669311aff7ebb6586234acfe4fe2e5698c9a3936249e09ddc782b1 2024-03-28T02:08:11.373 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:08:11 smithi178 systemd[1]: Started Ceph mds.l for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:11.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:11 smithi153 ceph-mon[36228]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:11.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:11 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] up:boot 2024-03-28T02:08:11.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:11 smithi153 ceph-mon[36228]: fsmap 1 up:standby 2024-03-28T02:08:11.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:11 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T02:08:11.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:11 smithi106 ceph-mon[31965]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:11.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:11 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] up:boot 2024-03-28T02:08:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:11 smithi106 ceph-mon[31965]: fsmap 1 up:standby 2024-03-28T02:08:11.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:11 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T02:08:12.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:12.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.231+0000 7f7f7fd97640 1 -- 172.21.15.106:0/877070523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 msgr2=0x7f7f78106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:12.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.231+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/877070523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 0x7f7f78106e60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7f68009900 tx=0x7f7f6802bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:12.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.231+0000 7f7f7fd97640 1 -- 172.21.15.106:0/877070523 shutdown_connections 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.231+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/877070523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f781073a0 0x7f7f781097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.231+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/877070523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 0x7f7f78106e60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.231+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/877070523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f78102140 0x7f7f78104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 -- 172.21.15.106:0/877070523 >> 172.21.15.106:0/877070523 conn(0x7f7f780fbb30 msgr2=0x7f7f780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 -- 172.21.15.106:0/877070523 shutdown_connections 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 -- 172.21.15.106:0/877070523 wait complete. 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 Processor -- start 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 -- start start 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f78102140 0x7f7f78197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 0x7f7f78197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f781073a0 0x7f7f7819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7810d330 con 0x7f7f78104a70 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7810d1b0 con 0x7f7f78102140 2024-03-28T02:08:12.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7fd97640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7810d4b0 con 0x7f7f781073a0 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7d332640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 0x7f7f78197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7d332640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 0x7f7f78197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:45512/0 (socket says 172.21.15.106:45512) 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7d332640 1 -- 172.21.15.106:0/1782291278 learned_addr learned my addr 172.21.15.106:0/1782291278 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.232+0000 7f7f7db33640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f78102140 0x7f7f78197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f7e334640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f781073a0 0x7f7f7819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f7d332640 1 -- 172.21.15.106:0/1782291278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f781073a0 msgr2=0x7f7f7819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f7d332640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f781073a0 0x7f7f7819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f7d332640 1 -- 172.21.15.106:0/1782291278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f78102140 msgr2=0x7f7f78197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f7d332640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f78102140 0x7f7f78197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f7d332640 1 -- 172.21.15.106:0/1782291278 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7819f620 con 0x7f7f78104a70 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f7d332640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 0x7f7f78197f40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7f68005ab0 tx=0x7f7f68005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f6effd640 1 -- 172.21.15.106:0/1782291278 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68041070 con 0x7f7f78104a70 2024-03-28T02:08:12.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f6effd640 1 -- 172.21.15.106:0/1782291278 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f680054e0 con 0x7f7f78104a70 2024-03-28T02:08:12.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7819f8b0 con 0x7f7f78104a70 2024-03-28T02:08:12.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f6effd640 1 -- 172.21.15.106:0/1782291278 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68034440 con 0x7f7f78104a70 2024-03-28T02:08:12.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.233+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7819fdf0 con 0x7f7f78104a70 2024-03-28T02:08:12.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.234+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f780fd620 con 0x7f7f78104a70 2024-03-28T02:08:12.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.235+0000 7f7f6effd640 1 -- 172.21.15.106:0/1782291278 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7f68044070 con 0x7f7f78104a70 2024-03-28T02:08:12.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.236+0000 7f7f6effd640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f640a2880 0x7f7f640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:12.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.236+0000 7f7f7db33640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f640a2880 0x7f7f640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:12.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.236+0000 7f7f6effd640 1 -- 172.21.15.106:0/1782291278 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7f680f0340 con 0x7f7f78104a70 2024-03-28T02:08:12.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.236+0000 7f7f7db33640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f640a2880 0x7f7f640a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f74005fd0 tx=0x7f7f74009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:12.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.239+0000 7f7f6effd640 1 -- 172.21.15.106:0/1782291278 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f680bd880 con 0x7f7f78104a70 2024-03-28T02:08:12.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.454+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f7802d230 con 0x7f7f78104a70 2024-03-28T02:08:12.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.454+0000 7f7f6effd640 1 -- 172.21.15.106:0/1782291278 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f7f680bd180 con 0x7f7f78104a70 2024-03-28T02:08:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:12.448 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T02:08:12.448 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 2 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f640a2880 msgr2=0x7f7f640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f640a2880 0x7f7f640a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f74005fd0 tx=0x7f7f74009680 comp rx=0 tx=0).stop 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 msgr2=0x7f7f78197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 0x7f7f78197f40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7f68005ab0 tx=0x7f7f68005dc0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 shutdown_connections 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f781073a0 0x7f7f7819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f78104a70 0x7f7f78197f40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f640a2880 0x7f7f640a4d40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 --2- 172.21.15.106:0/1782291278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f78102140 0x7f7f78197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 >> 172.21.15.106:0/1782291278 conn(0x7f7f780fbb30 msgr2=0x7f7f78102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:12.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 shutdown_connections 2024-03-28T02:08:12.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:12.456+0000 7f7f7fd97640 1 -- 172.21.15.106:0/1782291278 wait complete. 2024-03-28T02:08:12.459 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:12.459 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:12.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:12.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:12.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:12.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:12 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:12.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:12 smithi106 ceph-mon[31965]: Deploying daemon mds.k on smithi153 2024-03-28T02:08:12.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:12.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:12.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:12.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:12.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:12.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:12 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:12.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:12 smithi153 ceph-mon[36228]: Deploying daemon mds.k on smithi153 2024-03-28T02:08:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:12 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:12 smithi178 ceph-mon[36521]: Deploying daemon mds.k on smithi153 2024-03-28T02:08:13.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:13 smithi153 ceph-mon[36228]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:13.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:13 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1782291278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:13.483 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:08:13 smithi153 systemd[1]: Starting Ceph mds.k for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:13.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:13 smithi106 ceph-mon[31965]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:13.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:13 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1782291278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:13.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:13 smithi178 ceph-mon[36521]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:13.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:13 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1782291278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:13.858 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:08:13 smithi153 podman[66211]: 2024-03-28 02:08:13.502827572 +0000 UTC m=+0.018126991 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:13.859 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:08:13 smithi153 podman[66211]: 2024-03-28 02:08:13.604546681 +0000 UTC m=+0.119846111 container create 87cffb547bf8bdcb04087409f83d88ecd9fb85041d07f6a1ab53f0a639953292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-28T02:08:13.859 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:08:13 smithi153 podman[66211]: 2024-03-28 02:08:13.840593648 +0000 UTC m=+0.355893059 container init 87cffb547bf8bdcb04087409f83d88ecd9fb85041d07f6a1ab53f0a639953292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-03-28T02:08:13.859 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:08:13 smithi153 podman[66211]: 2024-03-28 02:08:13.843340183 +0000 UTC m=+0.358639598 container start 87cffb547bf8bdcb04087409f83d88ecd9fb85041d07f6a1ab53f0a639953292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T02:08:14.233 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:08:13 smithi153 ceph-mds[66230]: -- 172.21.15.153:0/3748782060 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eaead0fa40 con 0x55eaeace2c00 2024-03-28T02:08:14.233 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:08:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k[66226]: starting mds.k at 2024-03-28T02:08:14.233 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:08:13 smithi153 bash[66211]: 87cffb547bf8bdcb04087409f83d88ecd9fb85041d07f6a1ab53f0a639953292 2024-03-28T02:08:14.233 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:08:13 smithi153 systemd[1]: Started Ceph mds.k for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:14.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:14.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.578+0000 7fef283a9640 1 -- 172.21.15.106:0/2309743154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef201005c0 msgr2=0x7fef201009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:14.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.578+0000 7fef283a9640 1 --2- 172.21.15.106:0/2309743154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef201005c0 0x7fef201009e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009900 tx=0x7fef1c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:14.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.578+0000 7fef283a9640 1 -- 172.21.15.106:0/2309743154 shutdown_connections 2024-03-28T02:08:14.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.578+0000 7fef283a9640 1 --2- 172.21.15.106:0/2309743154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef20102170 0x7fef20108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.578+0000 7fef283a9640 1 --2- 172.21.15.106:0/2309743154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef201017b0 0x7fef20101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.578+0000 7fef283a9640 1 --2- 172.21.15.106:0/2309743154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef201005c0 0x7fef201009e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.578+0000 7fef283a9640 1 -- 172.21.15.106:0/2309743154 >> 172.21.15.106:0/2309743154 conn(0x7fef200fbb30 msgr2=0x7fef200fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:14.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.578+0000 7fef283a9640 1 -- 172.21.15.106:0/2309743154 shutdown_connections 2024-03-28T02:08:14.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.578+0000 7fef283a9640 1 -- 172.21.15.106:0/2309743154 wait complete. 2024-03-28T02:08:14.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef283a9640 1 Processor -- start 2024-03-28T02:08:14.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef283a9640 1 -- start start 2024-03-28T02:08:14.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef283a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef201005c0 0x7fef201968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:14.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef283a9640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef201017b0 0x7fef20196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:14.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef283a9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef20102170 0x7fef2019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:14.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef283a9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2010c210 con 0x7fef20102170 2024-03-28T02:08:14.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef283a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2010c090 con 0x7fef201005c0 2024-03-28T02:08:14.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef283a9640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2010c390 con 0x7fef201017b0 2024-03-28T02:08:14.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef26946640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef20102170 0x7fef2019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:14.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef26946640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef20102170 0x7fef2019de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:45534/0 (socket says 172.21.15.106:45534) 2024-03-28T02:08:14.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef26946640 1 -- 172.21.15.106:0/1538620393 learned_addr learned my addr 172.21.15.106:0/1538620393 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:14.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.579+0000 7fef26145640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef201005c0 0x7fef201968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef25944640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef201017b0 0x7fef20196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef26946640 1 -- 172.21.15.106:0/1538620393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef201017b0 msgr2=0x7fef20196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef26946640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef201017b0 0x7fef20196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef26946640 1 -- 172.21.15.106:0/1538620393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef201005c0 msgr2=0x7fef201968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef26946640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef201005c0 0x7fef201968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef26946640 1 -- 172.21.15.106:0/1538620393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2019e500 con 0x7fef20102170 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef26946640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef20102170 0x7fef2019de90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fef1800eaa0 tx=0x7fef1800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef177fe640 1 -- 172.21.15.106:0/1538620393 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1800cdb0 con 0x7fef20102170 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2019e7f0 con 0x7fef20102170 2024-03-28T02:08:14.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef177fe640 1 -- 172.21.15.106:0/1538620393 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef18005020 con 0x7fef20102170 2024-03-28T02:08:14.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef177fe640 1 -- 172.21.15.106:0/1538620393 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef180116b0 con 0x7fef20102170 2024-03-28T02:08:14.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.580+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2019ecf0 con 0x7fef20102170 2024-03-28T02:08:14.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.581+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef201079b0 con 0x7fef20102170 2024-03-28T02:08:14.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.582+0000 7fef177fe640 1 -- 172.21.15.106:0/1538620393 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fef18005560 con 0x7fef20102170 2024-03-28T02:08:14.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.583+0000 7fef177fe640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fef0c0a2800 0x7fef0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:14.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.583+0000 7fef177fe640 1 -- 172.21.15.106:0/1538620393 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fef180ce820 con 0x7fef20102170 2024-03-28T02:08:14.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.583+0000 7fef26145640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fef0c0a2800 0x7fef0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:14.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.583+0000 7fef26145640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fef0c0a2800 0x7fef0c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000c00 tx=0x7fef1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:14.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.588+0000 7fef177fe640 1 -- 172.21.15.106:0/1538620393 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fef18016030 con 0x7fef20102170 2024-03-28T02:08:14.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.829+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef2002d070 con 0x7fef20102170 2024-03-28T02:08:14.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.830+0000 7fef177fe640 1 -- 172.21.15.106:0/1538620393 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7fef1809bd60 con 0x7fef20102170 2024-03-28T02:08:14.824 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:14.824 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T02:08:14.824 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 3 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fef0c0a2800 msgr2=0x7fef0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fef0c0a2800 0x7fef0c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000c00 tx=0x7fef1c005fd0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef20102170 msgr2=0x7fef2019de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef20102170 0x7fef2019de90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fef1800eaa0 tx=0x7fef1800ef60 comp rx=0 tx=0).stop 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 shutdown_connections 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef20102170 0x7fef2019de90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef201017b0 0x7fef20196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fef0c0a2800 0x7fef0c0a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 --2- 172.21.15.106:0/1538620393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef201005c0 0x7fef201968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 >> 172.21.15.106:0/1538620393 conn(0x7fef200fbb30 msgr2=0x7fef20104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 shutdown_connections 2024-03-28T02:08:14.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:14.832+0000 7fef283a9640 1 -- 172.21.15.106:0/1538620393 wait complete. 2024-03-28T02:08:15.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:15.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:15.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:15.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:15.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:15.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:15.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: Deploying daemon mds.j on smithi106 2024-03-28T02:08:15.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6832/564442660,v1:172.21.15.153:6833/564442660] up:boot 2024-03-28T02:08:15.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: fsmap 2 up:standby 2024-03-28T02:08:15.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T02:08:15.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:15.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:14 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1538620393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:15.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:15.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:15.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:15.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:15.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:15.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:15.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: Deploying daemon mds.j on smithi106 2024-03-28T02:08:15.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6832/564442660,v1:172.21.15.153:6833/564442660] up:boot 2024-03-28T02:08:15.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: fsmap 2 up:standby 2024-03-28T02:08:15.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T02:08:15.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:15.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:14 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1538620393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:15.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:15.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:15.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:15.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:15.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:15.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:15.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: Deploying daemon mds.j on smithi106 2024-03-28T02:08:15.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6832/564442660,v1:172.21.15.153:6833/564442660] up:boot 2024-03-28T02:08:15.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: fsmap 2 up:standby 2024-03-28T02:08:15.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T02:08:15.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:15.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:14 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1538620393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:16.490 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:08:16 smithi106 systemd[1]: Starting Ceph mds.j for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:16.745 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:08:16 smithi106 podman[72643]: 2024-03-28 02:08:16.400559564 +0000 UTC m=+0.018375028 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:16.745 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:08:16 smithi106 podman[72643]: 2024-03-28 02:08:16.507981843 +0000 UTC m=+0.125797295 container create a89b9033c742fbc24f5212ba9f70bb9ebcf4a079c382327cd7c2da132681ba33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-j, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:08:16.745 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:08:16 smithi106 podman[72643]: 2024-03-28 02:08:16.752390408 +0000 UTC m=+0.370205866 container init a89b9033c742fbc24f5212ba9f70bb9ebcf4a079c382327cd7c2da132681ba33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T02:08:16.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 -- 172.21.15.106:0/2979536428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 msgr2=0x7f815c109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 --2- 172.21.15.106:0/2979536428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 0x7f815c109810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8154009ca0 tx=0x7f815402c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 -- 172.21.15.106:0/2979536428 shutdown_connections 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 --2- 172.21.15.106:0/2979536428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 0x7f815c109810 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 --2- 172.21.15.106:0/2979536428 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f815c104a90 0x7f815c106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 --2- 172.21.15.106:0/2979536428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f815c102160 0x7f815c104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 -- 172.21.15.106:0/2979536428 >> 172.21.15.106:0/2979536428 conn(0x7f815c0fbb50 msgr2=0x7f815c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 -- 172.21.15.106:0/2979536428 shutdown_connections 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 -- 172.21.15.106:0/2979536428 wait complete. 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.956+0000 7f8164063640 1 Processor -- start 2024-03-28T02:08:16.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8164063640 1 -- start start 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8164063640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f815c102160 0x7f815c1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8164063640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f815c104a90 0x7f815c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8164063640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 0x7f815c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8164063640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c10d360 con 0x7f815c1073c0 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8164063640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c10d1e0 con 0x7f815c102160 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8164063640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c10d4e0 con 0x7f815c104a90 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8162600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 0x7f815c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8161dff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f815c102160 0x7f815c1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f81615fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f815c104a90 0x7f815c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8162600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 0x7f815c19ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:45564/0 (socket says 172.21.15.106:45564) 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f81615fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f815c104a90 0x7f815c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:34028/0 (socket says 172.21.15.106:34028) 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8162600640 1 -- 172.21.15.106:0/2490152845 learned_addr learned my addr 172.21.15.106:0/2490152845 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8161dff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f815c102160 0x7f815c1979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:42168/0 (socket says 172.21.15.106:42168) 2024-03-28T02:08:16.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8162600640 1 -- 172.21.15.106:0/2490152845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f815c104a90 msgr2=0x7f815c197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:16.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8162600640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f815c104a90 0x7f815c197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:16.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8162600640 1 -- 172.21.15.106:0/2490152845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f815c102160 msgr2=0x7f815c1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:16.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8162600640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f815c102160 0x7f815c1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:16.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8162600640 1 -- 172.21.15.106:0/2490152845 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c19f5f0 con 0x7f815c1073c0 2024-03-28T02:08:16.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8161dff640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f815c102160 0x7f815c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:08:16.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.957+0000 7f8162600640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 0x7f815c19ef80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8154005cc0 tx=0x7f8154032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:16.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.959+0000 7f8152ffd640 1 -- 172.21.15.106:0/2490152845 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154041020 con 0x7f815c1073c0 2024-03-28T02:08:16.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.959+0000 7f8152ffd640 1 -- 172.21.15.106:0/2490152845 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8154034cd0 con 0x7f815c1073c0 2024-03-28T02:08:16.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.959+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815c19f880 con 0x7f815c1073c0 2024-03-28T02:08:16.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.959+0000 7f8152ffd640 1 -- 172.21.15.106:0/2490152845 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154039040 con 0x7f815c1073c0 2024-03-28T02:08:16.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.959+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815c19fd90 con 0x7f815c1073c0 2024-03-28T02:08:16.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.960+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815c0fd640 con 0x7f815c1073c0 2024-03-28T02:08:16.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.961+0000 7f8152ffd640 1 -- 172.21.15.106:0/2490152845 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8154044020 con 0x7f815c1073c0 2024-03-28T02:08:16.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.962+0000 7f8152ffd640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f81480a2880 0x7f81480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:16.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.962+0000 7f8152ffd640 1 -- 172.21.15.106:0/2490152845 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f81540f0380 con 0x7f815c1073c0 2024-03-28T02:08:16.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.962+0000 7f8161dff640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f81480a2880 0x7f81480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:16.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.962+0000 7f8161dff640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f81480a2880 0x7f81480a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8158009fc0 tx=0x7f8158005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:16.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:16.968+0000 7f8152ffd640 1 -- 172.21.15.106:0/2490152845 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f815404b020 con 0x7f815c1073c0 2024-03-28T02:08:17.083 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:08:16 smithi106 podman[72643]: 2024-03-28 02:08:16.755142943 +0000 UTC m=+0.372958405 container start a89b9033c742fbc24f5212ba9f70bb9ebcf4a079c382327cd7c2da132681ba33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-j, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T02:08:17.083 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:08:16 smithi106 ceph-mds[72662]: -- 172.21.15.106:0/2117669128 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5565367dba40 con 0x5565367af000 2024-03-28T02:08:17.083 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:08:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-j[72658]: starting mds.j at 2024-03-28T02:08:17.083 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:08:16 smithi106 bash[72643]: a89b9033c742fbc24f5212ba9f70bb9ebcf4a079c382327cd7c2da132681ba33 2024-03-28T02:08:17.083 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:08:16 smithi106 systemd[1]: Started Ceph mds.j for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:17.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.188+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f815c0622a0 con 0x7f815c1073c0 2024-03-28T02:08:17.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.189+0000 7f8152ffd640 1 -- 172.21.15.106:0/2490152845 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7f81540bd950 con 0x7f815c1073c0 2024-03-28T02:08:17.183 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:17.183 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T02:08:17.183 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 3 2024-03-28T02:08:17.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f81480a2880 msgr2=0x7f81480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:17.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f81480a2880 0x7f81480a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8158009fc0 tx=0x7f8158005f90 comp rx=0 tx=0).stop 2024-03-28T02:08:17.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 msgr2=0x7f815c19ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:17.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 0x7f815c19ef80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8154005cc0 tx=0x7f8154032f40 comp rx=0 tx=0).stop 2024-03-28T02:08:17.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 shutdown_connections 2024-03-28T02:08:17.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815c1073c0 0x7f815c19ef80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:17.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f815c104a90 0x7f815c197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:17.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f81480a2880 0x7f81480a4d40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:17.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 --2- 172.21.15.106:0/2490152845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f815c102160 0x7f815c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:17.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 >> 172.21.15.106:0/2490152845 conn(0x7f815c0fbb50 msgr2=0x7f815c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:17.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 shutdown_connections 2024-03-28T02:08:17.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:17.191+0000 7f8164063640 1 -- 172.21.15.106:0/2490152845 wait complete. 2024-03-28T02:08:17.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:17 smithi153 ceph-mon[36228]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:17.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:17.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:17.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:17.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:17.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:17.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:17 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:17.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:17 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2490152845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:17.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:17 smithi106 ceph-mon[31965]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:17.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:17.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:17.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:17.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:17.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:17.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:17 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:17.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:17 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2490152845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:17 smithi178 ceph-mon[36521]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:17 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:17.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:17 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2490152845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:18 smithi106 ceph-mon[31965]: Deploying daemon mds.d on smithi106 2024-03-28T02:08:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:18 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.106:6834/3072695267,v1:172.21.15.106:6835/3072695267] up:boot 2024-03-28T02:08:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:18 smithi106 ceph-mon[31965]: fsmap 3 up:standby 2024-03-28T02:08:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:18 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T02:08:18.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:18 smithi153 ceph-mon[36228]: Deploying daemon mds.d on smithi106 2024-03-28T02:08:18.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:18 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.106:6834/3072695267,v1:172.21.15.106:6835/3072695267] up:boot 2024-03-28T02:08:18.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:18 smithi153 ceph-mon[36228]: fsmap 3 up:standby 2024-03-28T02:08:18.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:18 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T02:08:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:18 smithi178 ceph-mon[36521]: Deploying daemon mds.d on smithi106 2024-03-28T02:08:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:18 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.106:6834/3072695267,v1:172.21.15.106:6835/3072695267] up:boot 2024-03-28T02:08:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:18 smithi178 ceph-mon[36521]: fsmap 3 up:standby 2024-03-28T02:08:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:18 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T02:08:19.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:19.225 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:08:19 smithi106 systemd[1]: Starting Ceph mds.d for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:19.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.314+0000 7f95eb6c0640 1 -- 172.21.15.106:0/3874483311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e40756e0 msgr2=0x7f95e4073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:19.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.314+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/3874483311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e40756e0 0x7f95e4073b30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f95dc009cd0 tx=0x7f95dc02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.314+0000 7f95eb6c0640 1 -- 172.21.15.106:0/3874483311 shutdown_connections 2024-03-28T02:08:19.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.314+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/3874483311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95e410dc90 0x7f95e4110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.314+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/3874483311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e4074100 0x7f95e4074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.314+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/3874483311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e40756e0 0x7f95e4073b30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.314+0000 7f95eb6c0640 1 -- 172.21.15.106:0/3874483311 >> 172.21.15.106:0/3874483311 conn(0x7f95e40fcc50 msgr2=0x7f95e40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:19.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.314+0000 7f95eb6c0640 1 -- 172.21.15.106:0/3874483311 shutdown_connections 2024-03-28T02:08:19.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.314+0000 7f95eb6c0640 1 -- 172.21.15.106:0/3874483311 wait complete. 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95eb6c0640 1 Processor -- start 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95eb6c0640 1 -- start start 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95eb6c0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95e4074100 0x7f95e419e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95eb6c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e40756e0 0x7f95e419e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95eb6c0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e410dc90 0x7f95e41a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95eb6c0640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4113910 con 0x7f95e410dc90 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95eb6c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4113790 con 0x7f95e40756e0 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95eb6c0640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4113a90 con 0x7f95e4074100 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e9c5d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e410dc90 0x7f95e41a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e9c5d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e410dc90 0x7f95e41a55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56536/0 (socket says 172.21.15.106:56536) 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e9c5d640 1 -- 172.21.15.106:0/1885352601 learned_addr learned my addr 172.21.15.106:0/1885352601 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e8c5b640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e40756e0 0x7f95e419e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:19.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e9c5d640 1 -- 172.21.15.106:0/1885352601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95e4074100 msgr2=0x7f95e419e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e9c5d640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95e4074100 0x7f95e419e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e9c5d640 1 -- 172.21.15.106:0/1885352601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e40756e0 msgr2=0x7f95e419e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e9c5d640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e40756e0 0x7f95e419e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e9c5d640 1 -- 172.21.15.106:0/1885352601 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e41a5c50 con 0x7f95e410dc90 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.315+0000 7f95e8c5b640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e40756e0 0x7f95e419e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.316+0000 7f95e9c5d640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e410dc90 0x7f95e41a55e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f95e000d950 tx=0x7f95e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.316+0000 7f95da7fc640 1 -- 172.21.15.106:0/1885352601 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0017030 con 0x7f95e410dc90 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.316+0000 7f95da7fc640 1 -- 172.21.15.106:0/1885352601 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95e0004020 con 0x7f95e410dc90 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.316+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e41a5f40 con 0x7f95e410dc90 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.316+0000 7f95da7fc640 1 -- 172.21.15.106:0/1885352601 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0003c00 con 0x7f95e410dc90 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.316+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e41a63b0 con 0x7f95e410dc90 2024-03-28T02:08:19.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.317+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e4103e60 con 0x7f95e410dc90 2024-03-28T02:08:19.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.318+0000 7f95da7fc640 1 -- 172.21.15.106:0/1885352601 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f95e001a070 con 0x7f95e410dc90 2024-03-28T02:08:19.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.318+0000 7f95da7fc640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f95d00a2800 0x7f95d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:19.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.318+0000 7f95e945c640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f95d00a2800 0x7f95d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:19.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.318+0000 7f95da7fc640 1 -- 172.21.15.106:0/1885352601 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f95e00cdf80 con 0x7f95e410dc90 2024-03-28T02:08:19.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.319+0000 7f95e945c640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f95d00a2800 0x7f95d00a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f95dc006010 tx=0x7f95dc0093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:19.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.323+0000 7f95da7fc640 1 -- 172.21.15.106:0/1885352601 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f95e0021070 con 0x7f95e410dc90 2024-03-28T02:08:19.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:19 smithi106 ceph-mon[31965]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:19.492 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:08:19 smithi106 podman[73009]: 2024-03-28 02:08:19.265219632 +0000 UTC m=+0.017590251 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:19.492 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:08:19 smithi106 podman[73009]: 2024-03-28 02:08:19.373113446 +0000 UTC m=+0.125484084 container create b97c63748bd24a9b0cf78e315a2d7b87df398d9a57d865614ab61099e93955e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:08:19.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.548+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95e402d070 con 0x7f95e410dc90 2024-03-28T02:08:19.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.548+0000 7f95da7fc640 1 -- 172.21.15.106:0/1885352601 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2913 (secure 0 0 0) 0x7f95e009b4c0 con 0x7f95e410dc90 2024-03-28T02:08:19.541 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:19.542 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T02:08:19.542 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 4 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f95d00a2800 msgr2=0x7f95d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f95d00a2800 0x7f95d00a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f95dc006010 tx=0x7f95dc0093f0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e410dc90 msgr2=0x7f95e41a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e410dc90 0x7f95e41a55e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f95e000d950 tx=0x7f95e000de10 comp rx=0 tx=0).stop 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 shutdown_connections 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95e410dc90 0x7f95e41a55e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f95d00a2800 0x7f95d00a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e40756e0 0x7f95e419e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 --2- 172.21.15.106:0/1885352601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95e4074100 0x7f95e419e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 >> 172.21.15.106:0/1885352601 conn(0x7f95e40fcc50 msgr2=0x7f95e4101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:19.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 shutdown_connections 2024-03-28T02:08:19.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:19.550+0000 7f95eb6c0640 1 -- 172.21.15.106:0/1885352601 wait complete. 2024-03-28T02:08:19.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:19 smithi153 ceph-mon[36228]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:19.833 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:08:19 smithi106 podman[73009]: 2024-03-28 02:08:19.61765657 +0000 UTC m=+0.370027166 container init b97c63748bd24a9b0cf78e315a2d7b87df398d9a57d865614ab61099e93955e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T02:08:19.833 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:08:19 smithi106 podman[73009]: 2024-03-28 02:08:19.620257227 +0000 UTC m=+0.372627827 container start b97c63748bd24a9b0cf78e315a2d7b87df398d9a57d865614ab61099e93955e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T02:08:19.833 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:08:19 smithi106 ceph-mds[73050]: -- 172.21.15.106:0/3022913662 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556207fb5a40 con 0x556207f89000 2024-03-28T02:08:19.833 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:08:19 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-d[73044]: starting mds.d at 2024-03-28T02:08:19.833 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:08:19 smithi106 bash[73009]: b97c63748bd24a9b0cf78e315a2d7b87df398d9a57d865614ab61099e93955e7 2024-03-28T02:08:19.833 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:08:19 smithi106 systemd[1]: Started Ceph mds.d for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:19 smithi178 ceph-mon[36521]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:20.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:20 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1885352601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:20.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:20.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:20.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:20.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:20.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:20.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:20 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:20 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1885352601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:20 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:20 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1885352601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:20 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:21.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:21.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.676+0000 7ff27797e640 1 -- 172.21.15.106:0/3240847409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2701028b0 msgr2=0x7ff270102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:21.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.676+0000 7ff27797e640 1 --2- 172.21.15.106:0/3240847409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2701028b0 0x7ff270102d30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009900 tx=0x7ff26c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:21.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.676+0000 7ff27797e640 1 -- 172.21.15.106:0/3240847409 shutdown_connections 2024-03-28T02:08:21.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.676+0000 7ff27797e640 1 --2- 172.21.15.106:0/3240847409 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff270103270 0x7ff270109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:21.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.676+0000 7ff27797e640 1 --2- 172.21.15.106:0/3240847409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2701028b0 0x7ff270102d30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.676+0000 7ff27797e640 1 --2- 172.21.15.106:0/3240847409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2701016c0 0x7ff270101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.676+0000 7ff27797e640 1 -- 172.21.15.106:0/3240847409 >> 172.21.15.106:0/3240847409 conn(0x7ff2700fcc50 msgr2=0x7ff2700ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.676+0000 7ff27797e640 1 -- 172.21.15.106:0/3240847409 shutdown_connections 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.676+0000 7ff27797e640 1 -- 172.21.15.106:0/3240847409 wait complete. 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff27797e640 1 Processor -- start 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff27797e640 1 -- start start 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff27797e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2701016c0 0x7ff2701979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff27797e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2701028b0 0x7ff270197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff27797e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff270103270 0x7ff27019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff27797e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27010d340 con 0x7ff270103270 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff27797e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27010d1c0 con 0x7ff2701028b0 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff27797e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27010d4c0 con 0x7ff2701016c0 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff275f1b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff270103270 0x7ff27019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff275f1b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff270103270 0x7ff27019ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56580/0 (socket says 172.21.15.106:56580) 2024-03-28T02:08:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff275f1b640 1 -- 172.21.15.106:0/180193456 learned_addr learned my addr 172.21.15.106:0/180193456 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff27571a640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2701016c0 0x7ff2701979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff274f19640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2701028b0 0x7ff270197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff275f1b640 1 -- 172.21.15.106:0/180193456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2701016c0 msgr2=0x7ff2701979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff275f1b640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2701016c0 0x7ff2701979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff275f1b640 1 -- 172.21.15.106:0/180193456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2701028b0 msgr2=0x7ff270197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff275f1b640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2701028b0 0x7ff270197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff275f1b640 1 -- 172.21.15.106:0/180193456 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27019f5e0 con 0x7ff270103270 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.677+0000 7ff275f1b640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff270103270 0x7ff27019ef70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff26800eaa0 tx=0x7ff26800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.678+0000 7ff2667fc640 1 -- 172.21.15.106:0/180193456 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26800cdb0 con 0x7ff270103270 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.678+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27019f8d0 con 0x7ff270103270 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.678+0000 7ff2667fc640 1 -- 172.21.15.106:0/180193456 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff268005020 con 0x7ff270103270 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.678+0000 7ff2667fc640 1 -- 172.21.15.106:0/180193456 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2680116b0 con 0x7ff270103270 2024-03-28T02:08:21.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.678+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27019fdf0 con 0x7ff270103270 2024-03-28T02:08:21.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.679+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2701089e0 con 0x7ff270103270 2024-03-28T02:08:21.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.680+0000 7ff2667fc640 1 -- 172.21.15.106:0/180193456 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff268005560 con 0x7ff270103270 2024-03-28T02:08:21.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.680+0000 7ff2667fc640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff25c0a2800 0x7ff25c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:21.675 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.681+0000 7ff27571a640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff25c0a2800 0x7ff25c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:21.675 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.681+0000 7ff2667fc640 1 -- 172.21.15.106:0/180193456 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff2680ce710 con 0x7ff270103270 2024-03-28T02:08:21.675 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.681+0000 7ff27571a640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff25c0a2800 0x7ff25c0a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff260005fd0 tx=0x7ff26000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:21.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.685+0000 7ff2667fc640 1 -- 172.21.15.106:0/180193456 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff268016030 con 0x7ff270103270 2024-03-28T02:08:21.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:21 smithi153 ceph-mon[36228]: Deploying daemon mds.e on smithi153 2024-03-28T02:08:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:21 smithi153 ceph-mon[36228]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:21 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.106:6836/497398504,v1:172.21.15.106:6837/497398504] up:boot 2024-03-28T02:08:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:21 smithi153 ceph-mon[36228]: fsmap 4 up:standby 2024-03-28T02:08:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:21 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T02:08:21.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:21 smithi106 ceph-mon[31965]: Deploying daemon mds.e on smithi153 2024-03-28T02:08:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:21 smithi106 ceph-mon[31965]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:21 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.106:6836/497398504,v1:172.21.15.106:6837/497398504] up:boot 2024-03-28T02:08:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:21 smithi106 ceph-mon[31965]: fsmap 4 up:standby 2024-03-28T02:08:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:21 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T02:08:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:21 smithi178 ceph-mon[36521]: Deploying daemon mds.e on smithi153 2024-03-28T02:08:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:21 smithi178 ceph-mon[36521]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:21 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.106:6836/497398504,v1:172.21.15.106:6837/497398504] up:boot 2024-03-28T02:08:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:21 smithi178 ceph-mon[36521]: fsmap 4 up:standby 2024-03-28T02:08:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:21 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T02:08:21.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.897+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff27002d070 con 0x7ff270103270 2024-03-28T02:08:21.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.897+0000 7ff2667fc640 1 -- 172.21.15.106:0/180193456 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3704 (secure 0 0 0) 0x7ff26809b600 con 0x7ff270103270 2024-03-28T02:08:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:21.891 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T02:08:21.891 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 5 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff25c0a2800 msgr2=0x7ff25c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff25c0a2800 0x7ff25c0a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff260005fd0 tx=0x7ff26000b410 comp rx=0 tx=0).stop 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff270103270 msgr2=0x7ff27019ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff270103270 0x7ff27019ef70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff26800eaa0 tx=0x7ff26800ef60 comp rx=0 tx=0).stop 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 shutdown_connections 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff270103270 0x7ff27019ef70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff25c0a2800 0x7ff25c0a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2701028b0 0x7ff270197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 --2- 172.21.15.106:0/180193456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2701016c0 0x7ff2701979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 >> 172.21.15.106:0/180193456 conn(0x7ff2700fcc50 msgr2=0x7ff270105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 shutdown_connections 2024-03-28T02:08:21.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:21.899+0000 7ff27797e640 1 -- 172.21.15.106:0/180193456 wait complete. 2024-03-28T02:08:22.145 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:08:21 smithi153 systemd[1]: Starting Ceph mds.e for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:22.413 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:08:22 smithi153 podman[66534]: 2024-03-28 02:08:22.055374692 +0000 UTC m=+0.017770346 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:22.413 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:08:22 smithi153 podman[66534]: 2024-03-28 02:08:22.159138757 +0000 UTC m=+0.121534388 container create a4218b5047f29f41a14e23aece7672d81f9cfaa951ab012c145d88cb886eef44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T02:08:22.413 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:08:22 smithi153 podman[66534]: 2024-03-28 02:08:22.420484346 +0000 UTC m=+0.382879993 container init a4218b5047f29f41a14e23aece7672d81f9cfaa951ab012c145d88cb886eef44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True) 2024-03-28T02:08:22.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:22 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/180193456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:22.733 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:08:22 smithi153 podman[66534]: 2024-03-28 02:08:22.423486804 +0000 UTC m=+0.385882452 container start a4218b5047f29f41a14e23aece7672d81f9cfaa951ab012c145d88cb886eef44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T02:08:22.733 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:08:22 smithi153 ceph-mds[66553]: -- 172.21.15.153:0/3221181369 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56502875da40 con 0x565028730c00 2024-03-28T02:08:22.733 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:08:22 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e[66549]: starting mds.e at 2024-03-28T02:08:22.733 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:08:22 smithi153 bash[66534]: a4218b5047f29f41a14e23aece7672d81f9cfaa951ab012c145d88cb886eef44 2024-03-28T02:08:22.733 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:08:22 smithi153 systemd[1]: Started Ceph mds.e for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:22.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:22 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/180193456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:22.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:22 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/180193456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:23.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:23.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:23.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:23.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:23.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:23.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:23.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:23.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] up:boot 2024-03-28T02:08:23.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: fsmap 5 up:standby 2024-03-28T02:08:23.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] up:boot 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: fsmap 5 up:standby 2024-03-28T02:08:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T02:08:23.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] up:boot 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: fsmap 5 up:standby 2024-03-28T02:08:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T02:08:24.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 -- 172.21.15.106:0/2168789767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 msgr2=0x7f38fc1009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:24.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2168789767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 0x7f38fc1009d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009ca0 tx=0x7f38ec02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 -- 172.21.15.106:0/2168789767 shutdown_connections 2024-03-28T02:08:24.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2168789767 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc102160 0x7f38fc1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2168789767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38fc1017a0 0x7f38fc101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2168789767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 0x7f38fc1009d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 -- 172.21.15.106:0/2168789767 >> 172.21.15.106:0/2168789767 conn(0x7f38fc0fbb40 msgr2=0x7f38fc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:24.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 -- 172.21.15.106:0/2168789767 shutdown_connections 2024-03-28T02:08:24.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 -- 172.21.15.106:0/2168789767 wait complete. 2024-03-28T02:08:24.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.023+0000 7f3901b0c640 1 Processor -- start 2024-03-28T02:08:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f3901b0c640 1 -- start start 2024-03-28T02:08:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f3901b0c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 0x7f38fc06a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f3901b0c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38fc1017a0 0x7f38fc19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f3901b0c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc102160 0x7f38fc1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f3901b0c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc10c200 con 0x7f38fc1005b0 2024-03-28T02:08:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f3901b0c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc10c080 con 0x7f38fc1017a0 2024-03-28T02:08:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f3901b0c640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc10c380 con 0x7f38fc102160 2024-03-28T02:08:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 0x7f38fc06a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 0x7f38fc06a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56604/0 (socket says 172.21.15.106:56604) 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fb7fe640 1 -- 172.21.15.106:0/2338290868 learned_addr learned my addr 172.21.15.106:0/2338290868 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fbfff640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc102160 0x7f38fc1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fb7fe640 1 -- 172.21.15.106:0/2338290868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc102160 msgr2=0x7f38fc1a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38faffd640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38fc1017a0 0x7f38fc19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fb7fe640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc102160 0x7f38fc1a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fb7fe640 1 -- 172.21.15.106:0/2338290868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38fc1017a0 msgr2=0x7f38fc19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fb7fe640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38fc1017a0 0x7f38fc19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fb7fe640 1 -- 172.21.15.106:0/2338290868 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc1a1890 con 0x7f38fc1005b0 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fbfff640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc102160 0x7f38fc1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38fb7fe640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 0x7f38fc06a880 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0324c0 tx=0x7f38ec032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38f8ff9640 1 -- 172.21.15.106:0/2338290868 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec039040 con 0x7f38fc1005b0 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f38f8ff9640 1 -- 172.21.15.106:0/2338290868 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ec044070 con 0x7f38fc1005b0 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.024+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38fc1a1ac0 con 0x7f38fc1005b0 2024-03-28T02:08:24.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.025+0000 7f38f8ff9640 1 -- 172.21.15.106:0/2338290868 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec0042a0 con 0x7f38fc1005b0 2024-03-28T02:08:24.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.025+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38fc1a1fb0 con 0x7f38fc1005b0 2024-03-28T02:08:24.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.026+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38fc1079a0 con 0x7f38fc1005b0 2024-03-28T02:08:24.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.027+0000 7f38f8ff9640 1 -- 172.21.15.106:0/2338290868 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f38ec003970 con 0x7f38fc1005b0 2024-03-28T02:08:24.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.027+0000 7f38f8ff9640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f38e80a2830 0x7f38e80a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:24.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.027+0000 7f38faffd640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f38e80a2830 0x7f38e80a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:24.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.027+0000 7f38f8ff9640 1 -- 172.21.15.106:0/2338290868 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f38ec0f0ab0 con 0x7f38fc1005b0 2024-03-28T02:08:24.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.028+0000 7f38faffd640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f38e80a2830 0x7f38e80a4cf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38e4005fd0 tx=0x7f38e4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:24.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.032+0000 7f38f8ff9640 1 -- 172.21.15.106:0/2338290868 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f38ec0be080 con 0x7f38fc1005b0 2024-03-28T02:08:24.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.244+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38fc02d070 con 0x7f38fc1005b0 2024-03-28T02:08:24.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.244+0000 7f38f8ff9640 1 -- 172.21.15.106:0/2338290868 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4498 (secure 0 0 0) 0x7f38ec0bd980 con 0x7f38fc1005b0 2024-03-28T02:08:24.237 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:24.238 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T02:08:24.238 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 6 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f38e80a2830 msgr2=0x7f38e80a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f38e80a2830 0x7f38e80a4cf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38e4005fd0 tx=0x7f38e4005850 comp rx=0 tx=0).stop 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 msgr2=0x7f38fc06a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 0x7f38fc06a880 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0324c0 tx=0x7f38ec032f20 comp rx=0 tx=0).stop 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 shutdown_connections 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc102160 0x7f38fc1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f38e80a2830 0x7f38e80a4cf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38fc1017a0 0x7f38fc19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 --2- 172.21.15.106:0/2338290868 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38fc1005b0 0x7f38fc06a880 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 >> 172.21.15.106:0/2338290868 conn(0x7f38fc0fbb40 msgr2=0x7f38fc104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 shutdown_connections 2024-03-28T02:08:24.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:24.246+0000 7f3901b0c640 1 -- 172.21.15.106:0/2338290868 wait complete. 2024-03-28T02:08:24.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:24 smithi106 ceph-mon[31965]: Deploying daemon mds.f on smithi178 2024-03-28T02:08:24.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:24 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2338290868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:24.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:24 smithi178 ceph-mon[36521]: Deploying daemon mds.f on smithi178 2024-03-28T02:08:24.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:24 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2338290868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:24.872 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:08:24 smithi178 systemd[1]: Starting Ceph mds.f for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:24 smithi153 ceph-mon[36228]: Deploying daemon mds.f on smithi178 2024-03-28T02:08:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:24 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2338290868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:25.372 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:08:24 smithi178 podman[67022]: 2024-03-28 02:08:24.840020193 +0000 UTC m=+0.018115475 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:25.372 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:08:24 smithi178 podman[67022]: 2024-03-28 02:08:24.954377807 +0000 UTC m=+0.132473082 container create e226f5b558f622d4981ef8bb458761dc4d94ca7bcda8cad49bba49b194fc9f8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:08:25.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:25 smithi106 ceph-mon[31965]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:25.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:25 smithi178 ceph-mon[36521]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:25.872 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:08:25 smithi178 podman[67022]: 2024-03-28 02:08:25.49081109 +0000 UTC m=+0.668906374 container init e226f5b558f622d4981ef8bb458761dc4d94ca7bcda8cad49bba49b194fc9f8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-28T02:08:25.872 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:08:25 smithi178 podman[67022]: 2024-03-28 02:08:25.493520702 +0000 UTC m=+0.671615985 container start e226f5b558f622d4981ef8bb458761dc4d94ca7bcda8cad49bba49b194fc9f8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T02:08:25.872 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:08:25 smithi178 ceph-mds[67041]: -- 172.21.15.178:0/875317911 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559cb7ad5a40 con 0x559cb7aa8800 2024-03-28T02:08:25.872 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:08:25 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f[67037]: starting mds.f at 2024-03-28T02:08:25.872 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:08:25 smithi178 bash[67022]: e226f5b558f622d4981ef8bb458761dc4d94ca7bcda8cad49bba49b194fc9f8f 2024-03-28T02:08:25.872 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:08:25 smithi178 systemd[1]: Started Ceph mds.f for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:25 smithi153 ceph-mon[36228]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:26.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:26.367 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 -- 172.21.15.106:0/2579165081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 msgr2=0x7f63dc1045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 --2- 172.21.15.106:0/2579165081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 0x7f63dc1045d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009900 tx=0x7f63d802bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 -- 172.21.15.106:0/2579165081 shutdown_connections 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 --2- 172.21.15.106:0/2579165081 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f63dc107440 0x7f63dc109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 --2- 172.21.15.106:0/2579165081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc104b10 0x7f63dc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 --2- 172.21.15.106:0/2579165081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 0x7f63dc1045d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 -- 172.21.15.106:0/2579165081 >> 172.21.15.106:0/2579165081 conn(0x7f63dc0fbb30 msgr2=0x7f63dc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 -- 172.21.15.106:0/2579165081 shutdown_connections 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 -- 172.21.15.106:0/2579165081 wait complete. 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 Processor -- start 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 -- start start 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 0x7f63dc1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f63dc104b10 0x7f63dc197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:26.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc107440 0x7f63dc19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc10d330 con 0x7f63dc1021e0 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc10d1b0 con 0x7f63dc107440 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.372+0000 7f63e3b10640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc10d4b0 con 0x7f63dc104b10 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e18ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 0x7f63dc1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e18ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 0x7f63dc1979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56620/0 (socket says 172.21.15.106:56620) 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e18ac640 1 -- 172.21.15.106:0/936307059 learned_addr learned my addr 172.21.15.106:0/936307059 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e10ab640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f63dc104b10 0x7f63dc197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e20ad640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc107440 0x7f63dc19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e18ac640 1 -- 172.21.15.106:0/936307059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f63dc104b10 msgr2=0x7f63dc197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e18ac640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f63dc104b10 0x7f63dc197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e18ac640 1 -- 172.21.15.106:0/936307059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc107440 msgr2=0x7f63dc19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e18ac640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc107440 0x7f63dc19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e18ac640 1 -- 172.21.15.106:0/936307059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc19f5d0 con 0x7f63dc1021e0 2024-03-28T02:08:26.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e18ac640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 0x7f63dc1979b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009230 tx=0x7f63d8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:26.370 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63d2ffd640 1 -- 172.21.15.106:0/936307059 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8041070 con 0x7f63dc1021e0 2024-03-28T02:08:26.370 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63dc19f860 con 0x7f63dc1021e0 2024-03-28T02:08:26.370 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63d2ffd640 1 -- 172.21.15.106:0/936307059 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63d80054e0 con 0x7f63dc1021e0 2024-03-28T02:08:26.370 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63d2ffd640 1 -- 172.21.15.106:0/936307059 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8034440 con 0x7f63dc1021e0 2024-03-28T02:08:26.370 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.373+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63dc19fda0 con 0x7f63dc1021e0 2024-03-28T02:08:26.370 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.374+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63dc0fd620 con 0x7f63dc1021e0 2024-03-28T02:08:26.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.375+0000 7f63d2ffd640 1 -- 172.21.15.106:0/936307059 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f63d8044070 con 0x7f63dc1021e0 2024-03-28T02:08:26.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.376+0000 7f63d2ffd640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f63c80a2800 0x7f63c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:26.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.376+0000 7f63e10ab640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f63c80a2800 0x7f63c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:26.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.376+0000 7f63d2ffd640 1 -- 172.21.15.106:0/936307059 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f63d80f02d0 con 0x7f63dc1021e0 2024-03-28T02:08:26.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.376+0000 7f63e10ab640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f63c80a2800 0x7f63c80a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f63dc198ec0 tx=0x7f63cc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:26.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.380+0000 7f63d2ffd640 1 -- 172.21.15.106:0/936307059 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f63d80bd810 con 0x7f63dc1021e0 2024-03-28T02:08:26.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.597+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63dc02d070 con 0x7f63dc1021e0 2024-03-28T02:08:26.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.597+0000 7f63d2ffd640 1 -- 172.21.15.106:0/936307059 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5292 (secure 0 0 0) 0x7f63d80bd110 con 0x7f63dc1021e0 2024-03-28T02:08:26.591 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:26.591 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-28T02:08:26.591 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 7 2024-03-28T02:08:26.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f63c80a2800 msgr2=0x7f63c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:26.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f63c80a2800 0x7f63c80a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f63dc198ec0 tx=0x7f63cc00b390 comp rx=0 tx=0).stop 2024-03-28T02:08:26.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 msgr2=0x7f63dc1979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:26.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 0x7f63dc1979b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009230 tx=0x7f63d8005dd0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 shutdown_connections 2024-03-28T02:08:26.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f63c80a2800 0x7f63c80a4cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc107440 0x7f63dc19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f63dc104b10 0x7f63dc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 --2- 172.21.15.106:0/936307059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63dc1021e0 0x7f63dc1979b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:26.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 >> 172.21.15.106:0/936307059 conn(0x7f63dc0fbb30 msgr2=0x7f63dc102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:26.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 shutdown_connections 2024-03-28T02:08:26.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:26.599+0000 7f63e3b10640 1 -- 172.21.15.106:0/936307059 wait complete. 2024-03-28T02:08:26.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] up:boot 2024-03-28T02:08:26.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: fsmap 6 up:standby 2024-03-28T02:08:26.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T02:08:26.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:26.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:26.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:26.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:26.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:26.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:26.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: Deploying daemon mds.c on smithi178 2024-03-28T02:08:26.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:26 smithi106 ceph-mon[31965]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] up:boot 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: fsmap 6 up:standby 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: Deploying daemon mds.c on smithi178 2024-03-28T02:08:26.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:26 smithi178 ceph-mon[36521]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] up:boot 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: fsmap 6 up:standby 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:26.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: Deploying daemon mds.c on smithi178 2024-03-28T02:08:26.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:26 smithi153 ceph-mon[36228]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:27.619 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:27 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/936307059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:27.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:27 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/936307059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:27.872 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:08:27 smithi178 systemd[1]: Starting Ceph mds.c for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:27 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/936307059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:28.372 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:08:28 smithi178 podman[67331]: 2024-03-28 02:08:27.92893783 +0000 UTC m=+0.017966960 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:28.372 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:08:28 smithi178 podman[67331]: 2024-03-28 02:08:28.058582584 +0000 UTC m=+0.147611746 container create a4878635d498d49025145ea64ef21042e34a71fb986aa60f506924128455837c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T02:08:28.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:28.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.719+0000 7f39e4df3640 1 -- 172.21.15.106:0/4169730792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 msgr2=0x7f39e0101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:28.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.719+0000 7f39e4df3640 1 --2- 172.21.15.106:0/4169730792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 0x7f39e0101c30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f39d4009ca0 tx=0x7f39d402c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.719+0000 7f39e4df3640 1 -- 172.21.15.106:0/4169730792 shutdown_connections 2024-03-28T02:08:28.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.719+0000 7f39e4df3640 1 --2- 172.21.15.106:0/4169730792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39e0102170 0x7f39e0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.719+0000 7f39e4df3640 1 --2- 172.21.15.106:0/4169730792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 0x7f39e0101c30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.719+0000 7f39e4df3640 1 --2- 172.21.15.106:0/4169730792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e01005c0 0x7f39e01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 -- 172.21.15.106:0/4169730792 >> 172.21.15.106:0/4169730792 conn(0x7f39e00fbb30 msgr2=0x7f39e00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:28.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 -- 172.21.15.106:0/4169730792 shutdown_connections 2024-03-28T02:08:28.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 -- 172.21.15.106:0/4169730792 wait complete. 2024-03-28T02:08:28.715 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 Processor -- start 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 -- start start 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39e01005c0 0x7f39e01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 0x7f39e0196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e0102170 0x7f39e019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e010c220 con 0x7f39e01017b0 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e010c0a0 con 0x7f39e0102170 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.720+0000 7f39e4df3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e010c3a0 con 0x7f39e01005c0 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ddd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 0x7f39e0196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39de59c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39e01005c0 0x7f39e01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ddd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 0x7f39e0196e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39766/0 (socket says 172.21.15.106:39766) 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ddd9b640 1 -- 172.21.15.106:0/1168571476 learned_addr learned my addr 172.21.15.106:0/1168571476 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:28.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ded9d640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e0102170 0x7f39e019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ddd9b640 1 -- 172.21.15.106:0/1168571476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39e01005c0 msgr2=0x7f39e01968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ddd9b640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39e01005c0 0x7f39e01968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ddd9b640 1 -- 172.21.15.106:0/1168571476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e0102170 msgr2=0x7f39e019de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ddd9b640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e0102170 0x7f39e019de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ddd9b640 1 -- 172.21.15.106:0/1168571476 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e019e4f0 con 0x7f39e01017b0 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39ddd9b640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 0x7f39e0196e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f39d40324c0 tx=0x7f39d4032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39cf7fe640 1 -- 172.21.15.106:0/1168571476 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d4039040 con 0x7f39e01017b0 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39cf7fe640 1 -- 172.21.15.106:0/1168571476 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39d4035c20 con 0x7f39e01017b0 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e019e780 con 0x7f39e01017b0 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39cf7fe640 1 -- 172.21.15.106:0/1168571476 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d4004310 con 0x7f39e01017b0 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.721+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e019ec70 con 0x7f39e01017b0 2024-03-28T02:08:28.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.722+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e01078e0 con 0x7f39e01017b0 2024-03-28T02:08:28.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.724+0000 7f39cf7fe640 1 -- 172.21.15.106:0/1168571476 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f39d4046030 con 0x7f39e01017b0 2024-03-28T02:08:28.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.724+0000 7f39cf7fe640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f39c80a2800 0x7f39c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:28.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.725+0000 7f39de59c640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f39c80a2800 0x7f39c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:28.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.725+0000 7f39cf7fe640 1 -- 172.21.15.106:0/1168571476 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f39d40f05c0 con 0x7f39e01017b0 2024-03-28T02:08:28.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.725+0000 7f39de59c640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f39c80a2800 0x7f39c80a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f39d8005fd0 tx=0x7f39d800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:28.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.728+0000 7f39cf7fe640 1 -- 172.21.15.106:0/1168571476 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f39d404d060 con 0x7f39e01017b0 2024-03-28T02:08:28.872 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:08:28 smithi178 podman[67331]: 2024-03-28 02:08:28.403276735 +0000 UTC m=+0.492305866 container init a4878635d498d49025145ea64ef21042e34a71fb986aa60f506924128455837c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T02:08:28.872 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:08:28 smithi178 podman[67331]: 2024-03-28 02:08:28.405926254 +0000 UTC m=+0.494955371 container start a4878635d498d49025145ea64ef21042e34a71fb986aa60f506924128455837c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=) 2024-03-28T02:08:28.872 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:08:28 smithi178 ceph-mds[67351]: -- 172.21.15.178:0/661562986 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564dc4f67a40 con 0x564dc4f3ac00 2024-03-28T02:08:28.872 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:08:28 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c[67347]: starting mds.c at 2024-03-28T02:08:28.872 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:08:28 smithi178 bash[67331]: a4878635d498d49025145ea64ef21042e34a71fb986aa60f506924128455837c 2024-03-28T02:08:28.872 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:08:28 smithi178 systemd[1]: Started Ceph mds.c for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:28.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.940+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39e002d2c0 con 0x7f39e01017b0 2024-03-28T02:08:28.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.941+0000 7f39cf7fe640 1 -- 172.21.15.106:0/1168571476 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6086 (secure 0 0 0) 0x7f39d4035dd0 con 0x7f39e01017b0 2024-03-28T02:08:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:28.934 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T02:08:28.935 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 8 2024-03-28T02:08:28.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f39c80a2800 msgr2=0x7f39c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:28.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f39c80a2800 0x7f39c80a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f39d8005fd0 tx=0x7f39d800b410 comp rx=0 tx=0).stop 2024-03-28T02:08:28.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 msgr2=0x7f39e0196e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:28.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 0x7f39e0196e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f39d40324c0 tx=0x7f39d4032f40 comp rx=0 tx=0).stop 2024-03-28T02:08:28.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 shutdown_connections 2024-03-28T02:08:28.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f39c80a2800 0x7f39c80a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e0102170 0x7f39e019de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39e01017b0 0x7f39e0196e10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 --2- 172.21.15.106:0/1168571476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39e01005c0 0x7f39e01968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:28.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 >> 172.21.15.106:0/1168571476 conn(0x7f39e00fbb30 msgr2=0x7f39e0104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:28.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 shutdown_connections 2024-03-28T02:08:28.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:28.943+0000 7f39e4df3640 1 -- 172.21.15.106:0/1168571476 wait complete. 2024-03-28T02:08:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:29.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:29.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6836/3554528514,v1:172.21.15.178:6837/3554528514] up:boot 2024-03-28T02:08:29.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: fsmap 7 up:standby 2024-03-28T02:08:29.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T02:08:29.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:29 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1168571476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6836/3554528514,v1:172.21.15.178:6837/3554528514] up:boot 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: fsmap 7 up:standby 2024-03-28T02:08:29.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T02:08:29.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:29 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1168571476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6836/3554528514,v1:172.21.15.178:6837/3554528514] up:boot 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: fsmap 7 up:standby 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T02:08:29.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:29 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1168571476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:30.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:30 smithi106 ceph-mon[31965]: Deploying daemon mds.h on smithi153 2024-03-28T02:08:30.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:30 smithi178 ceph-mon[36521]: Deploying daemon mds.h on smithi153 2024-03-28T02:08:30.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:30 smithi153 ceph-mon[36228]: Deploying daemon mds.h on smithi153 2024-03-28T02:08:30.944 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:08:30 smithi153 systemd[1]: Starting Ceph mds.h for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:30.949 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:31.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.068+0000 7f88d680b640 1 -- 172.21.15.106:0/520026438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 msgr2=0x7f88d0101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.068+0000 7f88d680b640 1 --2- 172.21.15.106:0/520026438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 0x7f88d0101c30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009900 tx=0x7f88c002bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.069+0000 7f88d680b640 1 -- 172.21.15.106:0/520026438 shutdown_connections 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.069+0000 7f88d680b640 1 --2- 172.21.15.106:0/520026438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d0102170 0x7f88d0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.069+0000 7f88d680b640 1 --2- 172.21.15.106:0/520026438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 0x7f88d0101c30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.069+0000 7f88d680b640 1 --2- 172.21.15.106:0/520026438 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88d01005c0 0x7f88d01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.069+0000 7f88d680b640 1 -- 172.21.15.106:0/520026438 >> 172.21.15.106:0/520026438 conn(0x7f88d00fbb30 msgr2=0x7f88d00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.069+0000 7f88d680b640 1 -- 172.21.15.106:0/520026438 shutdown_connections 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.069+0000 7f88d680b640 1 -- 172.21.15.106:0/520026438 wait complete. 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.069+0000 7f88d680b640 1 Processor -- start 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.069+0000 7f88d680b640 1 -- start start 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88d680b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88d01005c0 0x7f88d01968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:31.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88d680b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 0x7f88d0196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88d680b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d0102170 0x7f88d019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88d680b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d010c240 con 0x7f88d01017b0 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88d680b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d010c0c0 con 0x7f88d0102170 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88d680b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d010c3c0 con 0x7f88d01005c0 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cf7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 0x7f88d0196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cf7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 0x7f88d0196df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39786/0 (socket says 172.21.15.106:39786) 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cf7fe640 1 -- 172.21.15.106:0/3128666491 learned_addr learned my addr 172.21.15.106:0/3128666491 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cffff640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88d01005c0 0x7f88d01968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88d4da8640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d0102170 0x7f88d019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cf7fe640 1 -- 172.21.15.106:0/3128666491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88d01005c0 msgr2=0x7f88d01968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cf7fe640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88d01005c0 0x7f88d01968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cf7fe640 1 -- 172.21.15.106:0/3128666491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d0102170 msgr2=0x7f88d019de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:31.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cf7fe640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d0102170 0x7f88d019de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cf7fe640 1 -- 172.21.15.106:0/3128666491 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d019e4d0 con 0x7f88d01017b0 2024-03-28T02:08:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cf7fe640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 0x7f88d0196df0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009230 tx=0x7f88c0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cd7fa640 1 -- 172.21.15.106:0/3128666491 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0041070 con 0x7f88d01017b0 2024-03-28T02:08:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cd7fa640 1 -- 172.21.15.106:0/3128666491 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88c00044d0 con 0x7f88d01017b0 2024-03-28T02:08:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d019e760 con 0x7f88d01017b0 2024-03-28T02:08:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88cd7fa640 1 -- 172.21.15.106:0/3128666491 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0034440 con 0x7f88d01017b0 2024-03-28T02:08:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.070+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d019ec50 con 0x7f88d01017b0 2024-03-28T02:08:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.071+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d01078e0 con 0x7f88d01017b0 2024-03-28T02:08:31.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.072+0000 7f88cd7fa640 1 -- 172.21.15.106:0/3128666491 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f88c0044070 con 0x7f88d01017b0 2024-03-28T02:08:31.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.073+0000 7f88cd7fa640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88b80a2800 0x7f88b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:31.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.073+0000 7f88cffff640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88b80a2800 0x7f88b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:31.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.073+0000 7f88cd7fa640 1 -- 172.21.15.106:0/3128666491 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f88c00f0650 con 0x7f88d01017b0 2024-03-28T02:08:31.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.073+0000 7f88cffff640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88b80a2800 0x7f88b80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f88c4005fd0 tx=0x7f88c4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:31.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.076+0000 7f88cd7fa640 1 -- 172.21.15.106:0/3128666491 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f88c00bdb90 con 0x7f88d01017b0 2024-03-28T02:08:31.218 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:08:30 smithi153 podman[66853]: 2024-03-28 02:08:30.854319856 +0000 UTC m=+0.018030559 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:31.219 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:08:30 smithi153 podman[66853]: 2024-03-28 02:08:30.96470373 +0000 UTC m=+0.128414415 container create 3ebf9fb3013b026a66ec2cb23f2f517c10ba802577b61f011e23005d56639808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-03-28T02:08:31.219 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:08:31 smithi153 podman[66853]: 2024-03-28 02:08:31.225934561 +0000 UTC m=+0.389645250 container init 3ebf9fb3013b026a66ec2cb23f2f517c10ba802577b61f011e23005d56639808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T02:08:31.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.288+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88d002d070 con 0x7f88d01017b0 2024-03-28T02:08:31.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.289+0000 7f88cd7fa640 1 -- 172.21.15.106:0/3128666491 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6086 (secure 0 0 0) 0x7f88c00bd490 con 0x7f88d01017b0 2024-03-28T02:08:31.283 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:31.283 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T02:08:31.283 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 8 2024-03-28T02:08:31.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88b80a2800 msgr2=0x7f88b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:31.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88b80a2800 0x7f88b80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f88c4005fd0 tx=0x7f88c4005850 comp rx=0 tx=0).stop 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 msgr2=0x7f88d0196df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 0x7f88d0196df0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009230 tx=0x7f88c0005d50 comp rx=0 tx=0).stop 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 shutdown_connections 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88b80a2800 0x7f88b80a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d0102170 0x7f88d019de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88d01017b0 0x7f88d0196df0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 --2- 172.21.15.106:0/3128666491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88d01005c0 0x7f88d01968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 >> 172.21.15.106:0/3128666491 conn(0x7f88d00fbb30 msgr2=0x7f88d0104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 shutdown_connections 2024-03-28T02:08:31.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:31.291+0000 7f88d680b640 1 -- 172.21.15.106:0/3128666491 wait complete. 2024-03-28T02:08:31.483 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:08:31 smithi153 podman[66853]: 2024-03-28 02:08:31.228635709 +0000 UTC m=+0.392346398 container start 3ebf9fb3013b026a66ec2cb23f2f517c10ba802577b61f011e23005d56639808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T02:08:31.483 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-mds[66872]: -- 172.21.15.153:0/1170322711 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557a4ec4fa40 con 0x557a4ec22c00 2024-03-28T02:08:31.483 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h[66868]: starting mds.h at 2024-03-28T02:08:31.483 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:08:31 smithi153 bash[66853]: 3ebf9fb3013b026a66ec2cb23f2f517c10ba802577b61f011e23005d56639808 2024-03-28T02:08:31.483 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:08:31 smithi153 systemd[1]: Started Ceph mds.h for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:31 smithi106 ceph-mon[31965]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:31 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3128666491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:31.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:31.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:31 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:31.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:31 smithi178 ceph-mon[36521]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:31.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:31 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3128666491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:31.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:31.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:31.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:31 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:31.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-mon[36228]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:31.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3128666491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:31.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:31.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:31.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:31.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:31.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:31.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:31 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:32.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:32 smithi106 ceph-mon[31965]: Deploying daemon mds.i on smithi178 2024-03-28T02:08:32.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:32 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6836/1253460068,v1:172.21.15.153:6837/1253460068] up:boot 2024-03-28T02:08:32.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:32 smithi106 ceph-mon[31965]: fsmap 8 up:standby 2024-03-28T02:08:32.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:32 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T02:08:32.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:32 smithi178 ceph-mon[36521]: Deploying daemon mds.i on smithi178 2024-03-28T02:08:32.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:32 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6836/1253460068,v1:172.21.15.153:6837/1253460068] up:boot 2024-03-28T02:08:32.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:32 smithi178 ceph-mon[36521]: fsmap 8 up:standby 2024-03-28T02:08:32.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:32 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T02:08:32.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:32 smithi153 ceph-mon[36228]: Deploying daemon mds.i on smithi178 2024-03-28T02:08:32.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:32 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6836/1253460068,v1:172.21.15.153:6837/1253460068] up:boot 2024-03-28T02:08:32.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:32 smithi153 ceph-mon[36228]: fsmap 8 up:standby 2024-03-28T02:08:32.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:32 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T02:08:33.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 -- 172.21.15.106:0/2845338768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81017b0 msgr2=0x7f3fb8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/2845338768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81017b0 0x7f3fb8101c30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009900 tx=0x7f3fac02bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 -- 172.21.15.106:0/2845338768 shutdown_connections 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/2845338768 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fb8102170 0x7f3fb8108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/2845338768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81017b0 0x7f3fb8101c30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/2845338768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb81005c0 0x7f3fb81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 -- 172.21.15.106:0/2845338768 >> 172.21.15.106:0/2845338768 conn(0x7f3fb80fbb50 msgr2=0x7f3fb80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 -- 172.21.15.106:0/2845338768 shutdown_connections 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 -- 172.21.15.106:0/2845338768 wait complete. 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.416+0000 7f3fbd69a640 1 Processor -- start 2024-03-28T02:08:33.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fbd69a640 1 -- start start 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fbd69a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81005c0 0x7f3fb8104500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fbd69a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb81017b0 0x7f3fb8102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fbd69a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fb8102170 0x7f3fb81030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fbd69a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb810c240 con 0x7f3fb81005c0 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fbd69a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb810c0c0 con 0x7f3fb81017b0 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fbd69a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb810c3c0 con 0x7f3fb8102170 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81005c0 0x7f3fb8104500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81005c0 0x7f3fb8104500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39792/0 (socket says 172.21.15.106:39792) 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb6ffd640 1 -- 172.21.15.106:0/605253102 learned_addr learned my addr 172.21.15.106:0/605253102 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb77fe640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fb8102170 0x7f3fb81030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb67fc640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb81017b0 0x7f3fb8102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb6ffd640 1 -- 172.21.15.106:0/605253102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fb8102170 msgr2=0x7f3fb81030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb6ffd640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fb8102170 0x7f3fb81030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb6ffd640 1 -- 172.21.15.106:0/605253102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb81017b0 msgr2=0x7f3fb8102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb6ffd640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb81017b0 0x7f3fb8102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb6ffd640 1 -- 172.21.15.106:0/605253102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8103990 con 0x7f3fb81005c0 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb67fc640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb81017b0 0x7f3fb8102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fb6ffd640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81005c0 0x7f3fb8104500 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800d770 tx=0x7f3fa800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3f9ffff640 1 -- 172.21.15.106:0/605253102 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8017070 con 0x7f3fb81005c0 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb81a1b60 con 0x7f3fb81005c0 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3f9ffff640 1 -- 172.21.15.106:0/605253102 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fa8009c30 con 0x7f3fb81005c0 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.417+0000 7f3f9ffff640 1 -- 172.21.15.106:0/605253102 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8003dd0 con 0x7f3fb81005c0 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.418+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb81a2080 con 0x7f3fb81005c0 2024-03-28T02:08:33.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.419+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb81a2250 con 0x7f3fb81005c0 2024-03-28T02:08:33.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.420+0000 7f3f9ffff640 1 -- 172.21.15.106:0/605253102 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3fa8003f70 con 0x7f3fb81005c0 2024-03-28T02:08:33.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.421+0000 7f3f9ffff640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3fa00a2800 0x7f3fa00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:33.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.421+0000 7f3fb67fc640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3fa00a2800 0x7f3fa00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:33.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.421+0000 7f3f9ffff640 1 -- 172.21.15.106:0/605253102 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3fa80ce310 con 0x7f3fb81005c0 2024-03-28T02:08:33.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.421+0000 7f3fb67fc640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3fa00a2800 0x7f3fa00a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3fac036040 tx=0x7f3fac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:33.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.425+0000 7f3f9ffff640 1 -- 172.21.15.106:0/605253102 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3fa8009de0 con 0x7f3fb81005c0 2024-03-28T02:08:33.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:33 smithi106 ceph-mon[31965]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:33.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:33 smithi178 ceph-mon[36521]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:33.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.642+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fb81a25e0 con 0x7f3fb81005c0 2024-03-28T02:08:33.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.643+0000 7f3f9ffff640 1 -- 172.21.15.106:0/605253102 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6880 (secure 0 0 0) 0x7f3fa809b850 con 0x7f3fb81005c0 2024-03-28T02:08:33.636 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:33.636 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T02:08:33.637 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 9 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3fa00a2800 msgr2=0x7f3fa00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3fa00a2800 0x7f3fa00a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3fac036040 tx=0x7f3fac000bc0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81005c0 msgr2=0x7f3fb8104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81005c0 0x7f3fb8104500 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800d770 tx=0x7f3fa800dc30 comp rx=0 tx=0).stop 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 shutdown_connections 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fb8102170 0x7f3fb81030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3fa00a2800 0x7f3fa00a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb81017b0 0x7f3fb8102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 --2- 172.21.15.106:0/605253102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fb81005c0 0x7f3fb8104500 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 >> 172.21.15.106:0/605253102 conn(0x7f3fb80fbb50 msgr2=0x7f3fb80fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.645+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 shutdown_connections 2024-03-28T02:08:33.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:33.646+0000 7f3fbd69a640 1 -- 172.21.15.106:0/605253102 wait complete. 2024-03-28T02:08:33.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:33 smithi153 ceph-mon[36228]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:34.034 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:08:34 smithi178 podman[67641]: 2024-03-28 02:08:34.018756385 +0000 UTC m=+0.362210933 container start 5eb0e9cd985a7e6cb0675da502ef528f1d7023c0e2d1d2190055f5ae211f20b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T02:08:34.034 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:08:34 smithi178 ceph-mds[67660]: -- 172.21.15.178:0/327035076 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556b4bddba40 con 0x556b4bdaf000 2024-03-28T02:08:34.372 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:08:34 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i[67656]: starting mds.i at 2024-03-28T02:08:34.372 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:08:34 smithi178 bash[67641]: 5eb0e9cd985a7e6cb0675da502ef528f1d7023c0e2d1d2190055f5ae211f20b0 2024-03-28T02:08:34.372 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:08:34 smithi178 systemd[1]: Started Ceph mds.i for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:34.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:34 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/605253102' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:34.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:34 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:34.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:34 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:34.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:34 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:34.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:34 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:34.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:34 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:34.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:34 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:34.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:34 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/605253102' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:34 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:34 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:34 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:34 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:34 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:34 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:34.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:34 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/605253102' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:34.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:34 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:34.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:34 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:34.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:34 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:34.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:34 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:34.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:34 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:34.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:34 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:35.651 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:35.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.768+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/2720423537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 msgr2=0x7fdb18101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:35.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.768+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/2720423537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 0x7fdb18101c30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009900 tx=0x7fdb0c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:35.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.768+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/2720423537 shutdown_connections 2024-03-28T02:08:35.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.768+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/2720423537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb18102170 0x7fdb18108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.768+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/2720423537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 0x7fdb18101c30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.768+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/2720423537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb181005c0 0x7fdb181009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.768+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/2720423537 >> 172.21.15.106:0/2720423537 conn(0x7fdb180fbb30 msgr2=0x7fdb180fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:35.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.768+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/2720423537 shutdown_connections 2024-03-28T02:08:35.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.768+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/2720423537 wait complete. 2024-03-28T02:08:35.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb1e4b2640 1 Processor -- start 2024-03-28T02:08:35.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb1e4b2640 1 -- start start 2024-03-28T02:08:35.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb1e4b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb181005c0 0x7fdb181979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:35.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb1e4b2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 0x7fdb18197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:35.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb1e4b2640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb18102170 0x7fdb1819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:35.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb1e4b2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1810c240 con 0x7fdb181017b0 2024-03-28T02:08:35.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb1e4b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1810c0c0 con 0x7fdb181005c0 2024-03-28T02:08:35.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb1e4b2640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1810c3c0 con 0x7fdb18102170 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb177fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 0x7fdb18197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb177fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 0x7fdb18197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39820/0 (socket says 172.21.15.106:39820) 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb177fe640 1 -- 172.21.15.106:0/3584496945 learned_addr learned my addr 172.21.15.106:0/3584496945 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb1ca4f640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb18102170 0x7fdb1819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.769+0000 7fdb17fff640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb181005c0 0x7fdb181979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb177fe640 1 -- 172.21.15.106:0/3584496945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb18102170 msgr2=0x7fdb1819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb177fe640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb18102170 0x7fdb1819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb177fe640 1 -- 172.21.15.106:0/3584496945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb181005c0 msgr2=0x7fdb181979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb177fe640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb181005c0 0x7fdb181979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb177fe640 1 -- 172.21.15.106:0/3584496945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1819f600 con 0x7fdb181017b0 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb1ca4f640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb18102170 0x7fdb1819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb177fe640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 0x7fdb18197f20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009230 tx=0x7fdb0c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:35.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb157fa640 1 -- 172.21.15.106:0/3584496945 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c041070 con 0x7fdb181017b0 2024-03-28T02:08:35.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb157fa640 1 -- 172.21.15.106:0/3584496945 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb0c0054e0 con 0x7fdb181017b0 2024-03-28T02:08:35.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1819f890 con 0x7fdb181017b0 2024-03-28T02:08:35.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb157fa640 1 -- 172.21.15.106:0/3584496945 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c0346a0 con 0x7fdb181017b0 2024-03-28T02:08:35.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.770+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1819fd80 con 0x7fdb181017b0 2024-03-28T02:08:35.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.771+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1819ffe0 con 0x7fdb181017b0 2024-03-28T02:08:35.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.772+0000 7fdb157fa640 1 -- 172.21.15.106:0/3584496945 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdb0c044070 con 0x7fdb181017b0 2024-03-28T02:08:35.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.772+0000 7fdb157fa640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb000a2800 0x7fdb000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:35.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.773+0000 7fdb17fff640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb000a2800 0x7fdb000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:35.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.773+0000 7fdb157fa640 1 -- 172.21.15.106:0/3584496945 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdb0c0f03a0 con 0x7fdb181017b0 2024-03-28T02:08:35.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.773+0000 7fdb17fff640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb000a2800 0x7fdb000a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdb080058c0 tx=0x7fdb0800a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:35.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.778+0000 7fdb157fa640 1 -- 172.21.15.106:0/3584496945 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdb0c0bd8e0 con 0x7fdb181017b0 2024-03-28T02:08:35.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:35 smithi106 ceph-mon[31965]: Deploying daemon mds.g on smithi106 2024-03-28T02:08:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:35 smithi106 ceph-mon[31965]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:35 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] up:boot 2024-03-28T02:08:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:35 smithi106 ceph-mon[31965]: fsmap 9 up:standby 2024-03-28T02:08:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:35 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T02:08:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:35 smithi178 ceph-mon[36521]: Deploying daemon mds.g on smithi106 2024-03-28T02:08:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:35 smithi178 ceph-mon[36521]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:35 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] up:boot 2024-03-28T02:08:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:35 smithi178 ceph-mon[36521]: fsmap 9 up:standby 2024-03-28T02:08:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:35 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T02:08:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:35 smithi153 ceph-mon[36228]: Deploying daemon mds.g on smithi106 2024-03-28T02:08:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:35 smithi153 ceph-mon[36228]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:35 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] up:boot 2024-03-28T02:08:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:35 smithi153 ceph-mon[36228]: fsmap 9 up:standby 2024-03-28T02:08:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:35 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T02:08:35.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.996+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb181a01b0 con 0x7fdb181017b0 2024-03-28T02:08:35.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.997+0000 7fdb157fa640 1 -- 172.21.15.106:0/3584496945 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7673 (secure 0 0 0) 0x7fdb0c0bd1e0 con 0x7fdb181017b0 2024-03-28T02:08:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:35.990 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T02:08:35.990 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 10 2024-03-28T02:08:35.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb000a2800 msgr2=0x7fdb000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:35.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb000a2800 0x7fdb000a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdb080058c0 tx=0x7fdb0800a4c0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 msgr2=0x7fdb18197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:35.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 0x7fdb18197f20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009230 tx=0x7fdb0c004040 comp rx=0 tx=0).stop 2024-03-28T02:08:35.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 shutdown_connections 2024-03-28T02:08:35.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb18102170 0x7fdb1819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb181017b0 0x7fdb18197f20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fdb000a2800 0x7fdb000a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 --2- 172.21.15.106:0/3584496945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb181005c0 0x7fdb181979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:35.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 >> 172.21.15.106:0/3584496945 conn(0x7fdb180fbb30 msgr2=0x7fdb180fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:35.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 shutdown_connections 2024-03-28T02:08:35.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:35.999+0000 7fdb1e4b2640 1 -- 172.21.15.106:0/3584496945 wait complete. 2024-03-28T02:08:36.568 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:08:36 smithi106 systemd[1]: Starting Ceph mds.g for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:36.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:36 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3584496945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:36.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:36 smithi106 ceph-mon[31965]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:36.833 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:08:36 smithi106 podman[73637]: 2024-03-28 02:08:36.478484383 +0000 UTC m=+0.017875040 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:36.833 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:08:36 smithi106 podman[73637]: 2024-03-28 02:08:36.593492348 +0000 UTC m=+0.132882984 container create bce84fdd6b88f97c9e845c696d07bf2ab888f809d60e4aae14bf261aff56fef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=) 2024-03-28T02:08:36.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:36 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3584496945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:36.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:36 smithi178 ceph-mon[36521]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:36.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3584496945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:36.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:36 smithi153 ceph-mon[36228]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:37.333 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:08:36 smithi106 podman[73637]: 2024-03-28 02:08:36.874269302 +0000 UTC m=+0.413659936 container init bce84fdd6b88f97c9e845c696d07bf2ab888f809d60e4aae14bf261aff56fef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:08:37.333 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:08:36 smithi106 podman[73637]: 2024-03-28 02:08:36.877338372 +0000 UTC m=+0.416729006 container start bce84fdd6b88f97c9e845c696d07bf2ab888f809d60e4aae14bf261aff56fef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:08:37.333 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:08:36 smithi106 ceph-mds[73656]: -- 172.21.15.106:0/38865293 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5576a1117a40 con 0x5576a10eac00 2024-03-28T02:08:37.333 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:08:36 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g[73652]: starting mds.g at 2024-03-28T02:08:37.333 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:08:36 smithi106 bash[73637]: bce84fdd6b88f97c9e845c696d07bf2ab888f809d60e4aae14bf261aff56fef6 2024-03-28T02:08:37.333 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:08:36 smithi106 systemd[1]: Started Ceph mds.g for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:38.003 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:38.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.117+0000 7fcdd80b5640 1 -- 172.21.15.106:0/624841221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd01017b0 msgr2=0x7fcdd0101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:38.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.117+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/624841221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd01017b0 0x7fcdd0101c30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0009900 tx=0x7fcdc002bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 -- 172.21.15.106:0/624841221 shutdown_connections 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/624841221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd0102170 0x7fcdd0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/624841221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd01017b0 0x7fcdd0101c30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/624841221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd01005c0 0x7fcdd01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 -- 172.21.15.106:0/624841221 >> 172.21.15.106:0/624841221 conn(0x7fcdd00fbb30 msgr2=0x7fcdd00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 -- 172.21.15.106:0/624841221 shutdown_connections 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 -- 172.21.15.106:0/624841221 wait complete. 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 Processor -- start 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 -- start start 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd01005c0 0x7fcdd0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd01017b0 0x7fcdd0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd0102170 0x7fcdd019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:38.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd010d440 con 0x7fcdd0102170 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd010d2c0 con 0x7fcdd01017b0 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd80b5640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd010d5c0 con 0x7fcdd01005c0 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd6652640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd0102170 0x7fcdd019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd6652640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd0102170 0x7fcdd019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56024/0 (socket says 172.21.15.106:56024) 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.118+0000 7fcdd6652640 1 -- 172.21.15.106:0/1247282188 learned_addr learned my addr 172.21.15.106:0/1247282188 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd5650640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd01017b0 0x7fcdd0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd5e51640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd01005c0 0x7fcdd0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd6652640 1 -- 172.21.15.106:0/1247282188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd01005c0 msgr2=0x7fcdd0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd6652640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd01005c0 0x7fcdd0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd6652640 1 -- 172.21.15.106:0/1247282188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd01017b0 msgr2=0x7fcdd0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd6652640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd01017b0 0x7fcdd0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd6652640 1 -- 172.21.15.106:0/1247282188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd019f6d0 con 0x7fcdd0102170 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd5650640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd01017b0 0x7fcdd0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:08:38.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd5e51640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd01005c0 0x7fcdd0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:08:38.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd6652640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd0102170 0x7fcdd019f060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009fd0 tx=0x7fcdc800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:38.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdc6ffd640 1 -- 172.21.15.106:0/1247282188 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8011830 con 0x7fcdd0102170 2024-03-28T02:08:38.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd019f9c0 con 0x7fcdd0102170 2024-03-28T02:08:38.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdc6ffd640 1 -- 172.21.15.106:0/1247282188 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdc8016070 con 0x7fcdd0102170 2024-03-28T02:08:38.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdc6ffd640 1 -- 172.21.15.106:0/1247282188 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc800f2e0 con 0x7fcdd0102170 2024-03-28T02:08:38.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.119+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd019ff10 con 0x7fcdd0102170 2024-03-28T02:08:38.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.120+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd01078e0 con 0x7fcdd0102170 2024-03-28T02:08:38.116 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.121+0000 7fcdc6ffd640 1 -- 172.21.15.106:0/1247282188 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcdc80119d0 con 0x7fcdd0102170 2024-03-28T02:08:38.116 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.122+0000 7fcdc6ffd640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcdbc0a2800 0x7fcdbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:38.116 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.122+0000 7fcdd5e51640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcdbc0a2800 0x7fcdbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:38.116 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.122+0000 7fcdc6ffd640 1 -- 172.21.15.106:0/1247282188 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcdc80cdd10 con 0x7fcdd0102170 2024-03-28T02:08:38.116 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.122+0000 7fcdd5e51640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcdbc0a2800 0x7fcdbc0a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0101610 tx=0x7fcdcc0073c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:38.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.126+0000 7fcdc6ffd640 1 -- 172.21.15.106:0/1247282188 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcdc809b250 con 0x7fcdd0102170 2024-03-28T02:08:38.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:38.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:38.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:38.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:38.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:38.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:38.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: Deploying daemon mds.a on smithi106 2024-03-28T02:08:38.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.106:6838/1515774363,v1:172.21.15.106:6839/1515774363] up:boot 2024-03-28T02:08:38.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: fsmap 10 up:standby 2024-03-28T02:08:38.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:38 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T02:08:38.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.340+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdd002d070 con 0x7fcdd0102170 2024-03-28T02:08:38.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.341+0000 7fcdc6ffd640 1 -- 172.21.15.106:0/1247282188 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8468 (secure 0 0 0) 0x7fcdc809ab50 con 0x7fcdd0102170 2024-03-28T02:08:38.335 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:38.335 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T02:08:38.335 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 11 2024-03-28T02:08:38.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcdbc0a2800 msgr2=0x7fcdbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:38.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcdbc0a2800 0x7fcdbc0a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0101610 tx=0x7fcdcc0073c0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd0102170 msgr2=0x7fcdd019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:38.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd0102170 0x7fcdd019f060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009fd0 tx=0x7fcdc800b8e0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 shutdown_connections 2024-03-28T02:08:38.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdd0102170 0x7fcdd019f060 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcdbc0a2800 0x7fcdbc0a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd01017b0 0x7fcdd0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 --2- 172.21.15.106:0/1247282188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd01005c0 0x7fcdd0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:38.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 >> 172.21.15.106:0/1247282188 conn(0x7fcdd00fbb30 msgr2=0x7fcdd0104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:38.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 shutdown_connections 2024-03-28T02:08:38.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:38.344+0000 7fcdd80b5640 1 -- 172.21.15.106:0/1247282188 wait complete. 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: Deploying daemon mds.a on smithi106 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.106:6838/1515774363,v1:172.21.15.106:6839/1515774363] up:boot 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: fsmap 10 up:standby 2024-03-28T02:08:38.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:38 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T02:08:38.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:38.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:38.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:38.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:38.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:38.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:38.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: Deploying daemon mds.a on smithi106 2024-03-28T02:08:38.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.106:6838/1515774363,v1:172.21.15.106:6839/1515774363] up:boot 2024-03-28T02:08:38.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: fsmap 10 up:standby 2024-03-28T02:08:38.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:38 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T02:08:39.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:39 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1247282188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:39.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:39 smithi106 ceph-mon[31965]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:39.334 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:08:39 smithi106 systemd[1]: Starting Ceph mds.a for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:39.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:39 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1247282188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:39.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:39 smithi178 ceph-mon[36521]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:39.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:39 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1247282188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:39.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:39 smithi153 ceph-mon[36228]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:39.831 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:08:39 smithi106 podman[73983]: 2024-03-28 02:08:39.348157502 +0000 UTC m=+0.018351333 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:39.832 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:08:39 smithi106 podman[73983]: 2024-03-28 02:08:39.460911668 +0000 UTC m=+0.131105482 container create 26f52b2b47505a916bcc5e2cd0d8718fd3191479e8b9d190babe9beee450ea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:08:39.832 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:08:39 smithi106 podman[73983]: 2024-03-28 02:08:39.838879231 +0000 UTC m=+0.509073045 container init 26f52b2b47505a916bcc5e2cd0d8718fd3191479e8b9d190babe9beee450ea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-28T02:08:40.083 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:08:39 smithi106 podman[73983]: 2024-03-28 02:08:39.841720394 +0000 UTC m=+0.511914209 container start 26f52b2b47505a916bcc5e2cd0d8718fd3191479e8b9d190babe9beee450ea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T02:08:40.083 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:08:39 smithi106 ceph-mds[74002]: -- 172.21.15.106:0/2180375405 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5648951f1a40 con 0x5648951c4c00 2024-03-28T02:08:40.083 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:08:39 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a[73998]: starting mds.a at 2024-03-28T02:08:40.083 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:08:39 smithi106 bash[73983]: 26f52b2b47505a916bcc5e2cd0d8718fd3191479e8b9d190babe9beee450ea57 2024-03-28T02:08:40.083 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:08:39 smithi106 systemd[1]: Started Ceph mds.a for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:40.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:40.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.461+0000 7f19a580c640 1 -- 172.21.15.106:0/165067730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a0102170 msgr2=0x7f19a0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:40.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.461+0000 7f19a580c640 1 --2- 172.21.15.106:0/165067730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a0102170 0x7f19a0108a00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1998009ca0 tx=0x7f199802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.461+0000 7f19a580c640 1 -- 172.21.15.106:0/165067730 shutdown_connections 2024-03-28T02:08:40.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.461+0000 7f19a580c640 1 --2- 172.21.15.106:0/165067730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a0102170 0x7f19a0108a00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.461+0000 7f19a580c640 1 --2- 172.21.15.106:0/165067730 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19a01017b0 0x7f19a0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.461+0000 7f19a580c640 1 --2- 172.21.15.106:0/165067730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a01005c0 0x7f19a01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.461+0000 7f19a580c640 1 -- 172.21.15.106:0/165067730 >> 172.21.15.106:0/165067730 conn(0x7f19a00fbb30 msgr2=0x7f19a00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:40.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.461+0000 7f19a580c640 1 -- 172.21.15.106:0/165067730 shutdown_connections 2024-03-28T02:08:40.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.461+0000 7f19a580c640 1 -- 172.21.15.106:0/165067730 wait complete. 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19a580c640 1 Processor -- start 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19a580c640 1 -- start start 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19a580c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a01005c0 0x7f19a0069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19a580c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a01017b0 0x7f19a019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19a580c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19a0102170 0x7f19a019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19a580c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a010c240 con 0x7f19a01017b0 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19a580c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a010c0c0 con 0x7f19a01005c0 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19a580c640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a010c3c0 con 0x7f19a0102170 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19967fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a01017b0 0x7f19a019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19967fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a01017b0 0x7f19a019c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56078/0 (socket says 172.21.15.106:56078) 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f19967fc640 1 -- 172.21.15.106:0/3864272502 learned_addr learned my addr 172.21.15.106:0/3864272502 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f199f7fe640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19a0102170 0x7f19a019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.462+0000 7f199effd640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a01005c0 0x7f19a0069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19967fc640 1 -- 172.21.15.106:0/3864272502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19a0102170 msgr2=0x7f19a019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19967fc640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19a0102170 0x7f19a019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19967fc640 1 -- 172.21.15.106:0/3864272502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a01005c0 msgr2=0x7f19a0069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19967fc640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a01005c0 0x7f19a0069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19967fc640 1 -- 172.21.15.106:0/3864272502 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a019f5e0 con 0x7f19a01017b0 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19967fc640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a01017b0 0x7f19a019c6b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f198800e9f0 tx=0x7f198800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19a480a640 1 -- 172.21.15.106:0/3864272502 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198800cd80 con 0x7f19a01017b0 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19a480a640 1 -- 172.21.15.106:0/3864272502 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1988005500 con 0x7f19a01017b0 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a019f870 con 0x7f19a01017b0 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19a480a640 1 -- 172.21.15.106:0/3864272502 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1988010690 con 0x7f19a01017b0 2024-03-28T02:08:40.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.463+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a019fd70 con 0x7f19a01017b0 2024-03-28T02:08:40.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.464+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a0107880 con 0x7f19a01017b0 2024-03-28T02:08:40.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.465+0000 7f19a480a640 1 -- 172.21.15.106:0/3864272502 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f19880046c0 con 0x7f19a01017b0 2024-03-28T02:08:40.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.466+0000 7f19a480a640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f198c0a2800 0x7f198c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:40.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.466+0000 7f199effd640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f198c0a2800 0x7f198c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:40.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.466+0000 7f19a480a640 1 -- 172.21.15.106:0/3864272502 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f19880ceb90 con 0x7f19a01017b0 2024-03-28T02:08:40.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.466+0000 7f199effd640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f198c0a2800 0x7f198c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f19a0101610 tx=0x7f199000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:40.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.469+0000 7f19a480a640 1 -- 172.21.15.106:0/3864272502 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1988053740 con 0x7f19a01017b0 2024-03-28T02:08:40.675 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.682+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19a002d190 con 0x7f19a01017b0 2024-03-28T02:08:40.676 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.683+0000 7f19a480a640 1 -- 172.21.15.106:0/3864272502 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9263 (secure 0 0 0) 0x7f198809c0d0 con 0x7f19a01017b0 2024-03-28T02:08:40.676 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:40.676 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T02:08:40.676 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 12 2024-03-28T02:08:40.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.684+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f198c0a2800 msgr2=0x7f198c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:40.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.684+0000 7f19a580c640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f198c0a2800 0x7f198c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f19a0101610 tx=0x7f199000b390 comp rx=0 tx=0).stop 2024-03-28T02:08:40.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.684+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a01017b0 msgr2=0x7f19a019c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:40.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.684+0000 7f19a580c640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a01017b0 0x7f19a019c6b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f198800e9f0 tx=0x7f198800edf0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.685+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 shutdown_connections 2024-03-28T02:08:40.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.685+0000 7f19a580c640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19a0102170 0x7f19a019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.685+0000 7f19a580c640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a01017b0 0x7f19a019c6b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.685+0000 7f19a580c640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f198c0a2800 0x7f198c0a4cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.685+0000 7f19a580c640 1 --2- 172.21.15.106:0/3864272502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a01005c0 0x7f19a0069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:40.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.685+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 >> 172.21.15.106:0/3864272502 conn(0x7f19a00fbb30 msgr2=0x7f19a0104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:40.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.685+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 shutdown_connections 2024-03-28T02:08:40.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:40.685+0000 7f19a580c640 1 -- 172.21.15.106:0/3864272502 wait complete. 2024-03-28T02:08:41.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:41.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:41.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.106:6840/1766684450,v1:172.21.15.106:6841/1766684450] up:boot 2024-03-28T02:08:41.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: fsmap 11 up:standby 2024-03-28T02:08:41.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T02:08:41.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:41.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:41.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:41.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:41.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: Deploying daemon mds.b on smithi153 2024-03-28T02:08:41.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:41.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:41 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3864272502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.106:6840/1766684450,v1:172.21.15.106:6841/1766684450] up:boot 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: fsmap 11 up:standby 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: Deploying daemon mds.b on smithi153 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:41.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:41 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3864272502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.106:6840/1766684450,v1:172.21.15.106:6841/1766684450] up:boot 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: fsmap 11 up:standby 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:41.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: Deploying daemon mds.b on smithi153 2024-03-28T02:08:41.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:41.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:41 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3864272502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:42.483 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:42 smithi153 systemd[1]: Starting Ceph mds.b for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:08:42.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:42.741 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:42 smithi153 podman[67164]: 2024-03-28 02:08:42.399550025 +0000 UTC m=+0.018068484 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:08:42.741 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:42 smithi153 podman[67164]: 2024-03-28 02:08:42.503325894 +0000 UTC m=+0.121844354 container create 693b245038b96d31220ae81b52ec10c932fbec99d214619fb366808bb9cb6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-28T02:08:42.741 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:42 smithi153 podman[67164]: 2024-03-28 02:08:42.722632532 +0000 UTC m=+0.341150980 container init 693b245038b96d31220ae81b52ec10c932fbec99d214619fb366808bb9cb6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T02:08:42.741 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:42 smithi153 podman[67164]: 2024-03-28 02:08:42.725146074 +0000 UTC m=+0.343664516 container start 693b245038b96d31220ae81b52ec10c932fbec99d214619fb366808bb9cb6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:08:42.741 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:42 smithi153 ceph-mds[67183]: -- 172.21.15.153:0/1044215624 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55991bc8da40 con 0x55991bc60c00 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.821+0000 7f274d7f4640 1 -- 172.21.15.106:0/2068283324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27481005c0 msgr2=0x7f27481009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.821+0000 7f274d7f4640 1 --2- 172.21.15.106:0/2068283324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27481005c0 0x7f27481009e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2738009ca0 tx=0x7f273802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.821+0000 7f274d7f4640 1 -- 172.21.15.106:0/2068283324 shutdown_connections 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.821+0000 7f274d7f4640 1 --2- 172.21.15.106:0/2068283324 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2748102170 0x7f2748108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.821+0000 7f274d7f4640 1 --2- 172.21.15.106:0/2068283324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27481017b0 0x7f2748101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.821+0000 7f274d7f4640 1 --2- 172.21.15.106:0/2068283324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27481005c0 0x7f27481009e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.821+0000 7f274d7f4640 1 -- 172.21.15.106:0/2068283324 >> 172.21.15.106:0/2068283324 conn(0x7f27480fbb50 msgr2=0x7f27480fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.821+0000 7f274d7f4640 1 -- 172.21.15.106:0/2068283324 shutdown_connections 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.821+0000 7f274d7f4640 1 -- 172.21.15.106:0/2068283324 wait complete. 2024-03-28T02:08:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f274d7f4640 1 Processor -- start 2024-03-28T02:08:42.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f274d7f4640 1 -- start start 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f274d7f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27481005c0 0x7f27481979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f274d7f4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27481017b0 0x7f2748197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f274d7f4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2748102170 0x7f274819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f274d7f4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274810c230 con 0x7f27481017b0 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f274d7f4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274810c0b0 con 0x7f27481005c0 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f274d7f4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274810c3b0 con 0x7f2748102170 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f27467fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27481017b0 0x7f2748197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f27467fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27481017b0 0x7f2748197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56092/0 (socket says 172.21.15.106:56092) 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f27467fc640 1 -- 172.21.15.106:0/1659448172 learned_addr learned my addr 172.21.15.106:0/1659448172 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f2746ffd640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27481005c0 0x7f27481979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.822+0000 7f27477fe640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2748102170 0x7f274819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:42.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f2746ffd640 1 -- 172.21.15.106:0/1659448172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2748102170 msgr2=0x7f274819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f2746ffd640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2748102170 0x7f274819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f2746ffd640 1 -- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27481017b0 msgr2=0x7f2748197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f2746ffd640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27481017b0 0x7f2748197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f2746ffd640 1 -- 172.21.15.106:0/1659448172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274819f600 con 0x7f27481005c0 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f27477fe640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2748102170 0x7f274819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f27467fc640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27481017b0 0x7f2748197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f2746ffd640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27481005c0 0x7f27481979e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f273802c630 tx=0x7f2738032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f273ffff640 1 -- 172.21.15.106:0/1659448172 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738041020 con 0x7f27481005c0 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f273ffff640 1 -- 172.21.15.106:0/1659448172 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2738034cd0 con 0x7f27481005c0 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274819f890 con 0x7f27481005c0 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f273ffff640 1 -- 172.21.15.106:0/1659448172 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738039040 con 0x7f27481005c0 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.823+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274819fdd0 con 0x7f27481005c0 2024-03-28T02:08:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.824+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274819ffd0 con 0x7f27481005c0 2024-03-28T02:08:42.820 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.826+0000 7f273ffff640 1 -- 172.21.15.106:0/1659448172 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2738044020 con 0x7f27481005c0 2024-03-28T02:08:42.820 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.826+0000 7f273ffff640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27340a2880 0x7f27340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:42.820 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.826+0000 7f27467fc640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27340a2880 0x7f27340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:42.820 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.827+0000 7f273ffff640 1 -- 172.21.15.106:0/1659448172 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f27380f0790 con 0x7f27481005c0 2024-03-28T02:08:42.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.827+0000 7f27467fc640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27340a2880 0x7f27340a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2748198ef0 tx=0x7f2730005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:42.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:42.833+0000 7f273ffff640 1 -- 172.21.15.106:0/1659448172 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f273804b020 con 0x7f27481005c0 2024-03-28T02:08:43.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.058+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27481a01a0 con 0x7f27481005c0 2024-03-28T02:08:43.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.059+0000 7f273ffff640 1 -- 172.21.15.106:0/1659448172 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9263 (secure 0 0 0) 0x7f27380bdcd0 con 0x7f27481005c0 2024-03-28T02:08:43.052 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:43.052 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T02:08:43.053 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 12 2024-03-28T02:08:43.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27340a2880 msgr2=0x7f27340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27340a2880 0x7f27340a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2748198ef0 tx=0x7f2730005f90 comp rx=0 tx=0).stop 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27481005c0 msgr2=0x7f27481979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27481005c0 0x7f27481979e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f273802c630 tx=0x7f2738032f40 comp rx=0 tx=0).stop 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 shutdown_connections 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2748102170 0x7f274819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27481017b0 0x7f2748197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27340a2880 0x7f27340a4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 --2- 172.21.15.106:0/1659448172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27481005c0 0x7f27481979e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 >> 172.21.15.106:0/1659448172 conn(0x7f27480fbb50 msgr2=0x7f2748069cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 shutdown_connections 2024-03-28T02:08:43.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:43.061+0000 7f274d7f4640 1 -- 172.21.15.106:0/1659448172 wait complete. 2024-03-28T02:08:43.232 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:42 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b[67179]: starting mds.b at 2024-03-28T02:08:43.233 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:42 smithi153 bash[67164]: 693b245038b96d31220ae81b52ec10c932fbec99d214619fb366808bb9cb6a6c 2024-03-28T02:08:43.233 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:42 smithi153 systemd[1]: Started Ceph mds.b for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:08:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:43 smithi153 ceph-mon[36228]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:43 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1659448172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:08:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:43 smithi106 ceph-mon[31965]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:43 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1659448172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:08:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:43 smithi178 ceph-mon[36521]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:43 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1659448172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:08:44.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:44 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] up:boot 2024-03-28T02:08:44.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:44 smithi153 ceph-mon[36228]: fsmap 12 up:standby 2024-03-28T02:08:44.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:44 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T02:08:44.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:44 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] up:boot 2024-03-28T02:08:44.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:44 smithi178 ceph-mon[36521]: fsmap 12 up:standby 2024-03-28T02:08:44.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:44 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T02:08:44.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:44 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] up:boot 2024-03-28T02:08:44.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:44 smithi106 ceph-mon[31965]: fsmap 12 up:standby 2024-03-28T02:08:44.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:44 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T02:08:45.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:45.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.179+0000 7f9eec2dc640 1 -- 172.21.15.106:0/2029333640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4102160 msgr2=0x7f9ee4104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.179+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/2029333640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4102160 0x7f9ee4104550 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009900 tx=0x7f9ee002bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.179+0000 7f9eec2dc640 1 -- 172.21.15.106:0/2029333640 shutdown_connections 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.179+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/2029333640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ee41073c0 0x7f9ee4109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.179+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/2029333640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ee4104a90 0x7f9ee4106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.179+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/2029333640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4102160 0x7f9ee4104550 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.179+0000 7f9eec2dc640 1 -- 172.21.15.106:0/2029333640 >> 172.21.15.106:0/2029333640 conn(0x7f9ee40fbb50 msgr2=0x7f9ee40fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.179+0000 7f9eec2dc640 1 -- 172.21.15.106:0/2029333640 shutdown_connections 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.179+0000 7f9eec2dc640 1 -- 172.21.15.106:0/2029333640 wait complete. 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eec2dc640 1 Processor -- start 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eec2dc640 1 -- start start 2024-03-28T02:08:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eec2dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ee4102160 0x7f9ee4101ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eec2dc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4104a90 0x7f9ee4100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eec2dc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ee41073c0 0x7f9ee4100650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eec2dc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee410d370 con 0x7f9ee4104a90 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eec2dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee410d1f0 con 0x7f9ee4102160 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eec2dc640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee410d4f0 con 0x7f9ee41073c0 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9ee9877640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4104a90 0x7f9ee4100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9ee9877640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4104a90 0x7f9ee4100110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56116/0 (socket says 172.21.15.106:56116) 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9ee9877640 1 -- 172.21.15.106:0/820787417 learned_addr learned my addr 172.21.15.106:0/820787417 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eea879640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ee41073c0 0x7f9ee4100650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9eea078640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ee4102160 0x7f9ee4101ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9ee9877640 1 -- 172.21.15.106:0/820787417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ee41073c0 msgr2=0x7f9ee4100650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9ee9877640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ee41073c0 0x7f9ee4100650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9ee9877640 1 -- 172.21.15.106:0/820787417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ee4102160 msgr2=0x7f9ee4101ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9ee9877640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ee4102160 0x7f9ee4101ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.180+0000 7f9ee9877640 1 -- 172.21.15.106:0/820787417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee4100ee0 con 0x7f9ee4104a90 2024-03-28T02:08:45.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.181+0000 7f9ee9877640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4104a90 0x7f9ee4100110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4005fd0 tx=0x7f9ed400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:45.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.181+0000 7f9edb7fe640 1 -- 172.21.15.106:0/820787417 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4017070 con 0x7f9ee4104a90 2024-03-28T02:08:45.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.181+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee41a1b90 con 0x7f9ee4104a90 2024-03-28T02:08:45.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.181+0000 7f9edb7fe640 1 -- 172.21.15.106:0/820787417 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed4009c30 con 0x7f9ee4104a90 2024-03-28T02:08:45.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.181+0000 7f9edb7fe640 1 -- 172.21.15.106:0/820787417 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4010460 con 0x7f9ee4104a90 2024-03-28T02:08:45.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.181+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee41a2130 con 0x7f9ee4104a90 2024-03-28T02:08:45.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.182+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee40fd640 con 0x7f9ee4104a90 2024-03-28T02:08:45.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.183+0000 7f9edb7fe640 1 -- 172.21.15.106:0/820787417 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9ed4003e70 con 0x7f9ee4104a90 2024-03-28T02:08:45.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.183+0000 7f9edb7fe640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9ed00a2800 0x7f9ed00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.184+0000 7f9eea078640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9ed00a2800 0x7f9ed00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.184+0000 7f9edb7fe640 1 -- 172.21.15.106:0/820787417 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9ed40ce250 con 0x7f9ee4104a90 2024-03-28T02:08:45.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.184+0000 7f9eea078640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9ed00a2800 0x7f9ed00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009230 tx=0x7f9ee0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:45.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.188+0000 7f9edb7fe640 1 -- 172.21.15.106:0/820787417 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ed4020030 con 0x7f9ee4104a90 2024-03-28T02:08:45.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.412+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ee402d220 con 0x7f9ee4104a90 2024-03-28T02:08:45.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.413+0000 7f9edb7fe640 1 -- 172.21.15.106:0/820787417 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10055 (secure 0 0 0) 0x7f9ed409b790 con 0x7f9ee4104a90 2024-03-28T02:08:45.407 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:45.407 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-28T02:08:45.407 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 13 2024-03-28T02:08:45.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9ed00a2800 msgr2=0x7f9ed00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9ed00a2800 0x7f9ed00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009230 tx=0x7f9ee0005fd0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4104a90 msgr2=0x7f9ee4100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4104a90 0x7f9ee4100110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4005fd0 tx=0x7f9ed400dde0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 shutdown_connections 2024-03-28T02:08:45.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ee41073c0 0x7f9ee4100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ee4104a90 0x7f9ee4100110 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9ed00a2800 0x7f9ed00a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 --2- 172.21.15.106:0/820787417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ee4102160 0x7f9ee4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 >> 172.21.15.106:0/820787417 conn(0x7f9ee40fbb50 msgr2=0x7f9ee4102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:45.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 shutdown_connections 2024-03-28T02:08:45.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.416+0000 7f9eec2dc640 1 -- 172.21.15.106:0/820787417 wait complete. 2024-03-28T02:08:45.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:08:45.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.532+0000 7f77b56bf640 1 -- 172.21.15.106:0/701492500 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01005c0 msgr2=0x7f77b01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.532+0000 7f77b56bf640 1 --2- 172.21.15.106:0/701492500 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01005c0 0x7f77b01009e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009900 tx=0x7f77a002bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.532+0000 7f77b56bf640 1 -- 172.21.15.106:0/701492500 shutdown_connections 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.532+0000 7f77b56bf640 1 --2- 172.21.15.106:0/701492500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b0102170 0x7f77b0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.532+0000 7f77b56bf640 1 --2- 172.21.15.106:0/701492500 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77b01017b0 0x7f77b0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.532+0000 7f77b56bf640 1 --2- 172.21.15.106:0/701492500 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01005c0 0x7f77b01009e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.532+0000 7f77b56bf640 1 -- 172.21.15.106:0/701492500 >> 172.21.15.106:0/701492500 conn(0x7f77b00fbb30 msgr2=0x7f77b00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.532+0000 7f77b56bf640 1 -- 172.21.15.106:0/701492500 shutdown_connections 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.532+0000 7f77b56bf640 1 -- 172.21.15.106:0/701492500 wait complete. 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 Processor -- start 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 -- start start 2024-03-28T02:08:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77b01005c0 0x7f77b0069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01017b0 0x7f77b019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b0102170 0x7f77b019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b010c230 con 0x7f77b01017b0 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b010c0b0 con 0x7f77b0102170 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b010c3b0 con 0x7f77b01005c0 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77a7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01017b0 0x7f77b019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77a7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01017b0 0x7f77b019c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56132/0 (socket says 172.21.15.106:56132) 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77a7fff640 1 -- 172.21.15.106:0/4179577445 learned_addr learned my addr 172.21.15.106:0/4179577445 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77aeffd640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77b01005c0 0x7f77b0069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77af7fe640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b0102170 0x7f77b019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77a7fff640 1 -- 172.21.15.106:0/4179577445 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77b01005c0 msgr2=0x7f77b0069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77a7fff640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77b01005c0 0x7f77b0069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77a7fff640 1 -- 172.21.15.106:0/4179577445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b0102170 msgr2=0x7f77b019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77a7fff640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b0102170 0x7f77b019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77a7fff640 1 -- 172.21.15.106:0/4179577445 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b019f6b0 con 0x7f77b01017b0 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77af7fe640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b0102170 0x7f77b019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77a7fff640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01017b0 0x7f77b019c780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f779800d770 tx=0x7f779800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77acff9640 1 -- 172.21.15.106:0/4179577445 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7798017070 con 0x7f77b01017b0 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77acff9640 1 -- 172.21.15.106:0/4179577445 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7798005500 con 0x7f77b01017b0 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77acff9640 1 -- 172.21.15.106:0/4179577445 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7798004e00 con 0x7f77b01017b0 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b019f940 con 0x7f77b01017b0 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.533+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b019fe90 con 0x7f77b01017b0 2024-03-28T02:08:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.534+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b0107980 con 0x7f77b01017b0 2024-03-28T02:08:45.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.535+0000 7f77acff9640 1 -- 172.21.15.106:0/4179577445 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7798004fa0 con 0x7f77b01017b0 2024-03-28T02:08:45.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.536+0000 7f77acff9640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f779c0a2880 0x7f779c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.536+0000 7f77aeffd640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f779c0a2880 0x7f779c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.536+0000 7f77acff9640 1 -- 172.21.15.106:0/4179577445 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f77980cdf80 con 0x7f77b01017b0 2024-03-28T02:08:45.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.536+0000 7f77aeffd640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f779c0a2880 0x7f779c0a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f77a0000c00 tx=0x7f77a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:45.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.540+0000 7f77acff9640 1 -- 172.21.15.106:0/4179577445 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f779809b4c0 con 0x7f77b01017b0 2024-03-28T02:08:45.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.739+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77b002d2c0 con 0x7f77b01017b0 2024-03-28T02:08:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:45.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:08:45.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:45 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/820787417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:45.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.740+0000 7f77acff9640 1 -- 172.21.15.106:0/4179577445 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18534 (secure 0 0 0) 0x7f779809adc0 con 0x7f77b01017b0 2024-03-28T02:08:45.735 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:45.735 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":81,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:07:56.706770+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:08:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.742+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f779c0a2880 msgr2=0x7f779c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.742+0000 7f77b56bf640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f779c0a2880 0x7f779c0a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f77a0000c00 tx=0x7f77a0005fd0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.742+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01017b0 msgr2=0x7f77b019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.742+0000 7f77b56bf640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01017b0 0x7f77b019c780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f779800d770 tx=0x7f779800dc30 comp rx=0 tx=0).stop 2024-03-28T02:08:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.743+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 shutdown_connections 2024-03-28T02:08:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.743+0000 7f77b56bf640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f779c0a2880 0x7f779c0a4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.743+0000 7f77b56bf640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b0102170 0x7f77b019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.743+0000 7f77b56bf640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77b01017b0 0x7f77b019c780 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.743+0000 7f77b56bf640 1 --2- 172.21.15.106:0/4179577445 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77b01005c0 0x7f77b0069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.743+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 >> 172.21.15.106:0/4179577445 conn(0x7f77b00fbb30 msgr2=0x7f77b0104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:45.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.743+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 shutdown_connections 2024-03-28T02:08:45.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.743+0000 7f77b56bf640 1 -- 172.21.15.106:0/4179577445 wait complete. 2024-03-28T02:08:45.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:45.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:45.747 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:08:45.747 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.747 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:45 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/820787417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:45.748 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T02:08:45.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:08:45.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:45.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.802 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.802 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.802 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.802 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:45.802 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:08:45.802 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:45.802 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:45 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/820787417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:45.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 -- 172.21.15.106:0/4020600346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7364104b10 msgr2=0x7f7364106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 --2- 172.21.15.106:0/4020600346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7364104b10 0x7f7364106f00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7350009900 tx=0x7f735002bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:45.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 -- 172.21.15.106:0/4020600346 shutdown_connections 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 --2- 172.21.15.106:0/4020600346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7364107440 0x7f7364109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 --2- 172.21.15.106:0/4020600346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7364104b10 0x7f7364106f00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 --2- 172.21.15.106:0/4020600346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73641021e0 0x7f73641045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 -- 172.21.15.106:0/4020600346 >> 172.21.15.106:0/4020600346 conn(0x7f73640fbb30 msgr2=0x7f73640fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 -- 172.21.15.106:0/4020600346 shutdown_connections 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 -- 172.21.15.106:0/4020600346 wait complete. 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 Processor -- start 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.861+0000 7f736b244640 1 -- start start 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f736b244640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73641021e0 0x7f7364197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f736b244640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7364104b10 0x7f7364197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f736b244640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7364107440 0x7f736419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f736b244640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736410d320 con 0x7f73641021e0 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f736b244640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736410d1a0 con 0x7f7364104b10 2024-03-28T02:08:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f736b244640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736410d4a0 con 0x7f7364107440 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7368fe0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73641021e0 0x7f7364197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7368fe0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73641021e0 0x7f7364197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56162/0 (socket says 172.21.15.106:56162) 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7368fe0640 1 -- 172.21.15.106:0/3269999650 learned_addr learned my addr 172.21.15.106:0/3269999650 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7363fff640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7364104b10 0x7f7364197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f73697e1640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7364107440 0x7f736419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7368fe0640 1 -- 172.21.15.106:0/3269999650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7364107440 msgr2=0x7f736419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7368fe0640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7364107440 0x7f736419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7368fe0640 1 -- 172.21.15.106:0/3269999650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7364104b10 msgr2=0x7f7364197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7368fe0640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7364104b10 0x7f7364197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7368fe0640 1 -- 172.21.15.106:0/3269999650 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736419f650 con 0x7f73641021e0 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7363fff640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7364104b10 0x7f7364197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:08:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7368fe0640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73641021e0 0x7f7364197a30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f735800e9f0 tx=0x7f735800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:45.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7361ffb640 1 -- 172.21.15.106:0/3269999650 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735800cd80 con 0x7f73641021e0 2024-03-28T02:08:45.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7361ffb640 1 -- 172.21.15.106:0/3269999650 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7358005500 con 0x7f73641021e0 2024-03-28T02:08:45.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736419f940 con 0x7f73641021e0 2024-03-28T02:08:45.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f7361ffb640 1 -- 172.21.15.106:0/3269999650 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7358010690 con 0x7f73641021e0 2024-03-28T02:08:45.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.862+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736419fe10 con 0x7f73641021e0 2024-03-28T02:08:45.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.863+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73640fd620 con 0x7f73641021e0 2024-03-28T02:08:45.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.864+0000 7f7361ffb640 1 -- 172.21.15.106:0/3269999650 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f73580046c0 con 0x7f73641021e0 2024-03-28T02:08:45.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.865+0000 7f7361ffb640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f73540a2800 0x7f73540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:45.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.865+0000 7f7361ffb640 1 -- 172.21.15.106:0/3269999650 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f73580ce9b0 con 0x7f73641021e0 2024-03-28T02:08:45.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.865+0000 7f7363fff640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f73540a2800 0x7f73540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:45.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.865+0000 7f7363fff640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f73540a2800 0x7f73540a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7350000c00 tx=0x7f7350005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:45.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:45.869+0000 7f7361ffb640 1 -- 172.21.15.106:0/3269999650 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f735809bef0 con 0x7f73641021e0 2024-03-28T02:08:46.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.061+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f736402d070 con 0x7f73641021e0 2024-03-28T02:08:46.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.061+0000 7f7361ffb640 1 -- 172.21.15.106:0/3269999650 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f735809b7f0 con 0x7f73641021e0 2024-03-28T02:08:46.055 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:08:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f73540a2800 msgr2=0x7f73540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f73540a2800 0x7f73540a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7350000c00 tx=0x7f7350005fd0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73641021e0 msgr2=0x7f7364197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73641021e0 0x7f7364197a30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f735800e9f0 tx=0x7f735800edf0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 shutdown_connections 2024-03-28T02:08:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7364107440 0x7f736419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f73540a2800 0x7f73540a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7364104b10 0x7f7364197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 --2- 172.21.15.106:0/3269999650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73641021e0 0x7f7364197a30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 >> 172.21.15.106:0/3269999650 conn(0x7f73640fbb30 msgr2=0x7f7364102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:46.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 shutdown_connections 2024-03-28T02:08:46.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.063+0000 7f736b244640 1 -- 172.21.15.106:0/3269999650 wait complete. 2024-03-28T02:08:46.068 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T02:08:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.185+0000 7f98ce226640 1 -- 172.21.15.106:0/2753390968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c81073a0 msgr2=0x7f98c81097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.185+0000 7f98ce226640 1 --2- 172.21.15.106:0/2753390968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c81073a0 0x7f98c81097f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f98c0009ca0 tx=0x7f98c002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.185+0000 7f98ce226640 1 -- 172.21.15.106:0/2753390968 shutdown_connections 2024-03-28T02:08:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.185+0000 7f98ce226640 1 --2- 172.21.15.106:0/2753390968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c81073a0 0x7f98c81097f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.185+0000 7f98ce226640 1 --2- 172.21.15.106:0/2753390968 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8104a70 0x7f98c8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.185+0000 7f98ce226640 1 --2- 172.21.15.106:0/2753390968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c8102140 0x7f98c8104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.185+0000 7f98ce226640 1 -- 172.21.15.106:0/2753390968 >> 172.21.15.106:0/2753390968 conn(0x7f98c80fbb30 msgr2=0x7f98c80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.185+0000 7f98ce226640 1 -- 172.21.15.106:0/2753390968 shutdown_connections 2024-03-28T02:08:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.185+0000 7f98ce226640 1 -- 172.21.15.106:0/2753390968 wait complete. 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98ce226640 1 Processor -- start 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98ce226640 1 -- start start 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98ce226640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8102140 0x7f98c8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98ce226640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c8104a70 0x7f98c8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98ce226640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c81073a0 0x7f98c819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98ce226640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c810d430 con 0x7f98c8104a70 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98ce226640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c810d2b0 con 0x7f98c81073a0 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98ce226640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c810d5b0 con 0x7f98c8102140 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98c6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c8104a70 0x7f98c8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98c6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c8104a70 0x7f98c8197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56186/0 (socket says 172.21.15.106:56186) 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98c6ffd640 1 -- 172.21.15.106:0/2353050999 learned_addr learned my addr 172.21.15.106:0/2353050999 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98c77fe640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8102140 0x7f98c8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.186+0000 7f98c7fff640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c81073a0 0x7f98c819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98c6ffd640 1 -- 172.21.15.106:0/2353050999 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8102140 msgr2=0x7f98c8197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98c6ffd640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8102140 0x7f98c8197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98c6ffd640 1 -- 172.21.15.106:0/2353050999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c81073a0 msgr2=0x7f98c819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98c6ffd640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c81073a0 0x7f98c819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98c6ffd640 1 -- 172.21.15.106:0/2353050999 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c819f670 con 0x7f98c8104a70 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98c6ffd640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c8104a70 0x7f98c8197f90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b450 tx=0x7f98bc00b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98c4ff9640 1 -- 172.21.15.106:0/2353050999 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc015020 con 0x7f98c8104a70 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98c4ff9640 1 -- 172.21.15.106:0/2353050999 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98bc011d10 con 0x7f98c8104a70 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c819f960 con 0x7f98c8104a70 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98c4ff9640 1 -- 172.21.15.106:0/2353050999 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00f2e0 con 0x7f98c8104a70 2024-03-28T02:08:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.187+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c819ff30 con 0x7f98c8104a70 2024-03-28T02:08:46.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.188+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c80fd620 con 0x7f98c8104a70 2024-03-28T02:08:46.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.189+0000 7f98c4ff9640 1 -- 172.21.15.106:0/2353050999 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f98bc018020 con 0x7f98c8104a70 2024-03-28T02:08:46.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.190+0000 7f98c4ff9640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f98b00a2800 0x7f98b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:46.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.190+0000 7f98c77fe640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f98b00a2800 0x7f98b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:46.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.190+0000 7f98c4ff9640 1 -- 172.21.15.106:0/2353050999 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f98bc0cddc0 con 0x7f98c8104a70 2024-03-28T02:08:46.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.190+0000 7f98c77fe640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f98b00a2800 0x7f98b00a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009990 tx=0x7f98b8009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:46.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.194+0000 7f98c4ff9640 1 -- 172.21.15.106:0/2353050999 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f98bc01f020 con 0x7f98c8104a70 2024-03-28T02:08:46.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.379+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f98c802d2c0 con 0x7f98c8104a70 2024-03-28T02:08:46.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.379+0000 7f98c4ff9640 1 -- 172.21.15.106:0/2353050999 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f98bc09b300 con 0x7f98c8104a70 2024-03-28T02:08:46.373 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:46.373 INFO:teuthology.orchestra.run.smithi106.stdout:[ 2024-03-28T02:08:46.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:08:46.373 INFO:teuthology.orchestra.run.smithi106.stdout: "poolnum": 1, 2024-03-28T02:08:46.373 INFO:teuthology.orchestra.run.smithi106.stdout: "poolname": ".mgr" 2024-03-28T02:08:46.373 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:08:46.373 INFO:teuthology.orchestra.run.smithi106.stdout:] 2024-03-28T02:08:46.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f98b00a2800 msgr2=0x7f98b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f98b00a2800 0x7f98b00a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009990 tx=0x7f98b8009110 comp rx=0 tx=0).stop 2024-03-28T02:08:46.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c8104a70 msgr2=0x7f98c8197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c8104a70 0x7f98c8197f90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b450 tx=0x7f98bc00b850 comp rx=0 tx=0).stop 2024-03-28T02:08:46.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 shutdown_connections 2024-03-28T02:08:46.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f98b00a2800 0x7f98b00a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c81073a0 0x7f98c819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98c8104a70 0x7f98c8197f90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 --2- 172.21.15.106:0/2353050999 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8102140 0x7f98c8197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.381+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 >> 172.21.15.106:0/2353050999 conn(0x7f98c80fbb30 msgr2=0x7f98c8102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:46.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.382+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 shutdown_connections 2024-03-28T02:08:46.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.382+0000 7f98ce226640 1 -- 172.21.15.106:0/2353050999 wait complete. 2024-03-28T02:08:46.386 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T02:08:46.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T02:08:46.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 -- 172.21.15.106:0/3358276048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2541005a0 msgr2=0x7fa2541009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 --2- 172.21.15.106:0/3358276048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2541005a0 0x7fa2541009c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa244009900 tx=0x7fa24402bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:46.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 -- 172.21.15.106:0/3358276048 shutdown_connections 2024-03-28T02:08:46.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 --2- 172.21.15.106:0/3358276048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa254102150 0x7fa2541089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 --2- 172.21.15.106:0/3358276048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa254101790 0x7fa254101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 --2- 172.21.15.106:0/3358276048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2541005a0 0x7fa2541009c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 -- 172.21.15.106:0/3358276048 >> 172.21.15.106:0/3358276048 conn(0x7fa2540fbb30 msgr2=0x7fa2540fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:46.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 -- 172.21.15.106:0/3358276048 shutdown_connections 2024-03-28T02:08:46.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 -- 172.21.15.106:0/3358276048 wait complete. 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.504+0000 7fa259e59640 1 Processor -- start 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa259e59640 1 -- start start 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa259e59640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2541005a0 0x7fa254197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa259e59640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa254101790 0x7fa254197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa259e59640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa254102150 0x7fa25419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa259e59640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25410c200 con 0x7fa254101790 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa259e59640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25410c080 con 0x7fa2541005a0 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa259e59640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25410c380 con 0x7fa254102150 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa252ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa254101790 0x7fa254197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa252ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa254101790 0x7fa254197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56208/0 (socket says 172.21.15.106:56208) 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa252ffd640 1 -- 172.21.15.106:0/529315173 learned_addr learned my addr 172.21.15.106:0/529315173 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa2537fe640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2541005a0 0x7fa254197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:46.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa253fff640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa254102150 0x7fa25419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa252ffd640 1 -- 172.21.15.106:0/529315173 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa254102150 msgr2=0x7fa25419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa252ffd640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa254102150 0x7fa25419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa252ffd640 1 -- 172.21.15.106:0/529315173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2541005a0 msgr2=0x7fa254197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa252ffd640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2541005a0 0x7fa254197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa252ffd640 1 -- 172.21.15.106:0/529315173 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25419f670 con 0x7fa254101790 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa2537fe640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2541005a0 0x7fa254197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.505+0000 7fa252ffd640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa254101790 0x7fa254197f90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa24800e9f0 tx=0x7fa24800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.506+0000 7fa250ff9640 1 -- 172.21.15.106:0/529315173 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24800cd80 con 0x7fa254101790 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.506+0000 7fa250ff9640 1 -- 172.21.15.106:0/529315173 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa248010910 con 0x7fa254101790 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.506+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25419f960 con 0x7fa254101790 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.506+0000 7fa250ff9640 1 -- 172.21.15.106:0/529315173 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248018990 con 0x7fa254101790 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.506+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25419feb0 con 0x7fa254101790 2024-03-28T02:08:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.507+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2541a0140 con 0x7fa254101790 2024-03-28T02:08:46.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.508+0000 7fa250ff9640 1 -- 172.21.15.106:0/529315173 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa248010ac0 con 0x7fa254101790 2024-03-28T02:08:46.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.508+0000 7fa250ff9640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:46.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.508+0000 7fa2537fe640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:46.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.508+0000 7fa250ff9640 1 -- 172.21.15.106:0/529315173 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa2480ce9d0 con 0x7fa254101790 2024-03-28T02:08:46.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.509+0000 7fa2537fe640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa24402c7b0 tx=0x7fa244005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:46.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.513+0000 7fa250ff9640 1 -- 172.21.15.106:0/529315173 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa24809bf10 con 0x7fa254101790 2024-03-28T02:08:46.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:46.691+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa2541a0310 con 0x7fa254101790 2024-03-28T02:08:46.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:46 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:08:46.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:46 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4179577445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:46.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:46 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3269999650' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:08:46.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:46 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2353050999' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T02:08:46.778 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:46 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:08:46.779 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:46 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4179577445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:46.779 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:46 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3269999650' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:08:46.779 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:46 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2353050999' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T02:08:46.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:46 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:08:46.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:46 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4179577445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:46.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:46 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3269999650' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:08:46.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:46 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2353050999' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T02:08:47.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.429+0000 7fa250ff9640 1 -- 172.21.15.106:0/529315173 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa24809b810 con 0x7fa254101790 2024-03-28T02:08:47.422 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'cephfs_metadata' created 2024-03-28T02:08:47.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa23c0a2800 msgr2=0x7fa23c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:47.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa24402c7b0 tx=0x7fa244005fd0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa254101790 msgr2=0x7fa254197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:47.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa254101790 0x7fa254197f90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa24800e9f0 tx=0x7fa24800eeb0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.425 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 shutdown_connections 2024-03-28T02:08:47.425 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa254102150 0x7fa25419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.425 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa254101790 0x7fa254197f90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.425 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.425 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 --2- 172.21.15.106:0/529315173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2541005a0 0x7fa254197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.425 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.431+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 >> 172.21.15.106:0/529315173 conn(0x7fa2540fbb30 msgr2=0x7fa254068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:47.425 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.432+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 shutdown_connections 2024-03-28T02:08:47.425 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.432+0000 7fa259e59640 1 -- 172.21.15.106:0/529315173 wait complete. 2024-03-28T02:08:47.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-28T02:08:47.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.556+0000 7f5588e6c640 1 -- 172.21.15.106:0/344852643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 msgr2=0x7f5584102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:47.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.556+0000 7f5588e6c640 1 --2- 172.21.15.106:0/344852643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 0x7f5584102d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f556c009900 tx=0x7f556c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:47.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.556+0000 7f5588e6c640 1 -- 172.21.15.106:0/344852643 shutdown_connections 2024-03-28T02:08:47.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.556+0000 7f5588e6c640 1 --2- 172.21.15.106:0/344852643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5584103280 0x7f5584109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.556+0000 7f5588e6c640 1 --2- 172.21.15.106:0/344852643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 0x7f5584102d40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.556+0000 7f5588e6c640 1 --2- 172.21.15.106:0/344852643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55841016d0 0x7f5584101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.556+0000 7f5588e6c640 1 -- 172.21.15.106:0/344852643 >> 172.21.15.106:0/344852643 conn(0x7f55840fcc40 msgr2=0x7f55840ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:47.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.556+0000 7f5588e6c640 1 -- 172.21.15.106:0/344852643 shutdown_connections 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.556+0000 7f5588e6c640 1 -- 172.21.15.106:0/344852643 wait complete. 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f5588e6c640 1 Processor -- start 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f5588e6c640 1 -- start start 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f5588e6c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55841016d0 0x7f55841979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f5588e6c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 0x7f5584197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f5588e6c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5584103280 0x7f558419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f5588e6c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558410d350 con 0x7f55841028c0 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f5588e6c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558410d1d0 con 0x7f5584103280 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f5588e6c640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558410d4d0 con 0x7f55841016d0 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f557bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 0x7f5584197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:47.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f557bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 0x7f5584197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53082/0 (socket says 172.21.15.106:53082) 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f557bfff640 1 -- 172.21.15.106:0/3582471848 learned_addr learned my addr 172.21.15.106:0/3582471848 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f558259c640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55841016d0 0x7f55841979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f5582d9d640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5584103280 0x7f558419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f557bfff640 1 -- 172.21.15.106:0/3582471848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55841016d0 msgr2=0x7f55841979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f557bfff640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55841016d0 0x7f55841979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f557bfff640 1 -- 172.21.15.106:0/3582471848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5584103280 msgr2=0x7f558419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f557bfff640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5584103280 0x7f558419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f557bfff640 1 -- 172.21.15.106:0/3582471848 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558419f5f0 con 0x7f55841028c0 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.557+0000 7f557bfff640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 0x7f5584197f10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f556c009230 tx=0x7f556c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.558+0000 7f557b7fe640 1 -- 172.21.15.106:0/3582471848 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c041070 con 0x7f55841028c0 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.558+0000 7f557b7fe640 1 -- 172.21.15.106:0/3582471848 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f556c004450 con 0x7f55841028c0 2024-03-28T02:08:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.558+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558419f880 con 0x7f55841028c0 2024-03-28T02:08:47.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.558+0000 7f557b7fe640 1 -- 172.21.15.106:0/3582471848 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c003e20 con 0x7f55841028c0 2024-03-28T02:08:47.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.558+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558419fd70 con 0x7f55841028c0 2024-03-28T02:08:47.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.559+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55841089f0 con 0x7f55841028c0 2024-03-28T02:08:47.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.560+0000 7f557b7fe640 1 -- 172.21.15.106:0/3582471848 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f556c044070 con 0x7f55841028c0 2024-03-28T02:08:47.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.561+0000 7f557b7fe640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f55700a2800 0x7f55700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:47.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.561+0000 7f558259c640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f55700a2800 0x7f55700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:47.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.561+0000 7f557b7fe640 1 -- 172.21.15.106:0/3582471848 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f556c0f06d0 con 0x7f55841028c0 2024-03-28T02:08:47.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.561+0000 7f558259c640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f55700a2800 0x7f55700a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5584102720 tx=0x7f5574005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:47.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.565+0000 7f557b7fe640 1 -- 172.21.15.106:0/3582471848 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f556c0bd3b0 con 0x7f55841028c0 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/529315173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:47.740+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f558402d070 con 0x7f55841028c0 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/529315173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/529315173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:47.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:08:48.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.431+0000 7f557b7fe640 1 -- 172.21.15.106:0/3582471848 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f556c0c1450 con 0x7f55841028c0 2024-03-28T02:08:48.424 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'cephfs_data' created 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f55700a2800 msgr2=0x7f55700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f55700a2800 0x7f55700a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5584102720 tx=0x7f5574005f90 comp rx=0 tx=0).stop 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 msgr2=0x7f5584197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 0x7f5584197f10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f556c009230 tx=0x7f556c02c650 comp rx=0 tx=0).stop 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 shutdown_connections 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f55700a2800 0x7f55700a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5584103280 0x7f558419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55841028c0 0x7f5584197f10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 --2- 172.21.15.106:0/3582471848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55841016d0 0x7f55841979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.433+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 >> 172.21.15.106:0/3582471848 conn(0x7f55840fcc40 msgr2=0x7f5584105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:48.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.434+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 shutdown_connections 2024-03-28T02:08:48.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.434+0000 7f5588e6c640 1 -- 172.21.15.106:0/3582471848 wait complete. 2024-03-28T02:08:48.446 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T02:08:48.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.559+0000 7f6b75590640 1 -- 172.21.15.106:0/2155380517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70102120 msgr2=0x7f6b70104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.559+0000 7f6b75590640 1 --2- 172.21.15.106:0/2155380517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70102120 0x7f6b70104510 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009ca0 tx=0x7f6b6002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.559+0000 7f6b75590640 1 -- 172.21.15.106:0/2155380517 shutdown_connections 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.559+0000 7f6b75590640 1 --2- 172.21.15.106:0/2155380517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b70107380 0x7f6b701097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.559+0000 7f6b75590640 1 --2- 172.21.15.106:0/2155380517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b70104a50 0x7f6b70106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.559+0000 7f6b75590640 1 --2- 172.21.15.106:0/2155380517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70102120 0x7f6b70104510 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.559+0000 7f6b75590640 1 -- 172.21.15.106:0/2155380517 >> 172.21.15.106:0/2155380517 conn(0x7f6b700fbb30 msgr2=0x7f6b700fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 -- 172.21.15.106:0/2155380517 shutdown_connections 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 -- 172.21.15.106:0/2155380517 wait complete. 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 Processor -- start 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 -- start start 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b70102120 0x7f6b70197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:48.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b70104a50 0x7f6b70197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70107380 0x7f6b7019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7010d320 con 0x7f6b70107380 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7010d1a0 con 0x7f6b70102120 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b75590640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7010d4a0 con 0x7f6b70104a50 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b6f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70107380 0x7f6b7019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b6f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70107380 0x7f6b7019efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53110/0 (socket says 172.21.15.106:53110) 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.560+0000 7f6b6f7fe640 1 -- 172.21.15.106:0/3675300232 learned_addr learned my addr 172.21.15.106:0/3675300232 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b6effd640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b70102120 0x7f6b70197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b6e7fc640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b70104a50 0x7f6b70197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b6f7fe640 1 -- 172.21.15.106:0/3675300232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b70104a50 msgr2=0x7f6b70197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b6f7fe640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b70104a50 0x7f6b70197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b6f7fe640 1 -- 172.21.15.106:0/3675300232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b70102120 msgr2=0x7f6b70197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b6f7fe640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b70102120 0x7f6b70197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:48.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b6f7fe640 1 -- 172.21.15.106:0/3675300232 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7019f640 con 0x7f6b70107380 2024-03-28T02:08:48.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b6e7fc640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b70104a50 0x7f6b70197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:08:48.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b6f7fe640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70107380 0x7f6b7019efd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800d950 tx=0x7f6b6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:48.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b57fff640 1 -- 172.21.15.106:0/3675300232 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68017030 con 0x7f6b70107380 2024-03-28T02:08:48.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7019f930 con 0x7f6b70107380 2024-03-28T02:08:48.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b57fff640 1 -- 172.21.15.106:0/3675300232 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b68004020 con 0x7f6b70107380 2024-03-28T02:08:48.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b57fff640 1 -- 172.21.15.106:0/3675300232 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68003c00 con 0x7f6b70107380 2024-03-28T02:08:48.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.561+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7019fe80 con 0x7f6b70107380 2024-03-28T02:08:48.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.562+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b700fd620 con 0x7f6b70107380 2024-03-28T02:08:48.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.563+0000 7f6b57fff640 1 -- 172.21.15.106:0/3675300232 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6b6801a070 con 0x7f6b70107380 2024-03-28T02:08:48.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.564+0000 7f6b57fff640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6b580a2800 0x7f6b580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:48.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.564+0000 7f6b6effd640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6b580a2800 0x7f6b580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:48.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.564+0000 7f6b57fff640 1 -- 172.21.15.106:0/3675300232 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f6b680ce480 con 0x7f6b70107380 2024-03-28T02:08:48.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.564+0000 7f6b6effd640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6b580a2800 0x7f6b580a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6b60006010 tx=0x7f6b60032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:48.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.567+0000 7f6b57fff640 1 -- 172.21.15.106:0/3675300232 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b68021070 con 0x7f6b70107380 2024-03-28T02:08:48.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:48 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/529315173' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T02:08:48.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:48 smithi153 ceph-mon[36228]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T02:08:48.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:48 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3582471848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T02:08:48.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:48.780+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6b7002d070 con 0x7f6b70107380 2024-03-28T02:08:48.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:48 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/529315173' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T02:08:48.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:48 smithi106 ceph-mon[31965]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T02:08:48.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:48 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3582471848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T02:08:48.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:48 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/529315173' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T02:08:48.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:48 smithi178 ceph-mon[36521]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T02:08:48.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:48 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3582471848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T02:08:49.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.443+0000 7f6b57fff640 1 -- 172.21.15.106:0/3675300232 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f6b6809b6d0 con 0x7f6b70107380 2024-03-28T02:08:49.437 INFO:teuthology.orchestra.run.smithi106.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6b580a2800 msgr2=0x7f6b580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6b580a2800 0x7f6b580a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6b60006010 tx=0x7f6b60032590 comp rx=0 tx=0).stop 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70107380 msgr2=0x7f6b7019efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70107380 0x7f6b7019efd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800d950 tx=0x7f6b6800de10 comp rx=0 tx=0).stop 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 shutdown_connections 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b70107380 0x7f6b7019efd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b70104a50 0x7f6b70197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6b580a2800 0x7f6b580a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 --2- 172.21.15.106:0/3675300232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b70102120 0x7f6b70197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 >> 172.21.15.106:0/3675300232 conn(0x7f6b700fbb30 msgr2=0x7f6b70102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 shutdown_connections 2024-03-28T02:08:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.446+0000 7f6b75590640 1 -- 172.21.15.106:0/3675300232 wait complete. 2024-03-28T02:08:49.455 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-28T02:08:49.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-28T02:08:49.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.572+0000 7f1463d1b640 1 -- 172.21.15.106:0/663066295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c107380 msgr2=0x7f145c1097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.572+0000 7f1463d1b640 1 --2- 172.21.15.106:0/663066295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c107380 0x7f145c1097d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1454009ca0 tx=0x7f145402c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.572+0000 7f1463d1b640 1 -- 172.21.15.106:0/663066295 shutdown_connections 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.572+0000 7f1463d1b640 1 --2- 172.21.15.106:0/663066295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c107380 0x7f145c1097d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.572+0000 7f1463d1b640 1 --2- 172.21.15.106:0/663066295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f145c104a50 0x7f145c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.572+0000 7f1463d1b640 1 --2- 172.21.15.106:0/663066295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f145c102120 0x7f145c104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.572+0000 7f1463d1b640 1 -- 172.21.15.106:0/663066295 >> 172.21.15.106:0/663066295 conn(0x7f145c0fbb30 msgr2=0x7f145c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.572+0000 7f1463d1b640 1 -- 172.21.15.106:0/663066295 shutdown_connections 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.572+0000 7f1463d1b640 1 -- 172.21.15.106:0/663066295 wait complete. 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1463d1b640 1 Processor -- start 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1463d1b640 1 -- start start 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1463d1b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c102120 0x7f145c101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1463d1b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f145c104a50 0x7f145c1000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1463d1b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f145c107380 0x7f145c100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1463d1b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c10d390 con 0x7f145c102120 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1463d1b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c10d210 con 0x7f145c107380 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1463d1b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c10d510 con 0x7f145c104a50 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1461ab7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c102120 0x7f145c101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1461ab7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c102120 0x7f145c101a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53118/0 (socket says 172.21.15.106:53118) 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1461ab7640 1 -- 172.21.15.106:0/1390007671 learned_addr learned my addr 172.21.15.106:0/1390007671 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f14622b8640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f145c107380 0x7f145c100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f14612b6640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f145c104a50 0x7f145c1000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1461ab7640 1 -- 172.21.15.106:0/1390007671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f145c104a50 msgr2=0x7f145c1000d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1461ab7640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f145c104a50 0x7f145c1000d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1461ab7640 1 -- 172.21.15.106:0/1390007671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f145c107380 msgr2=0x7f145c100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1461ab7640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f145c107380 0x7f145c100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1461ab7640 1 -- 172.21.15.106:0/1390007671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c100ea0 con 0x7f145c102120 2024-03-28T02:08:49.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1461ab7640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c102120 0x7f145c101a90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f145800c890 tx=0x7f145800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:49.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1452ffd640 1 -- 172.21.15.106:0/1390007671 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458004220 con 0x7f145c102120 2024-03-28T02:08:49.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.573+0000 7f1452ffd640 1 -- 172.21.15.106:0/1390007671 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14580043c0 con 0x7f145c102120 2024-03-28T02:08:49.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.574+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f145c1a1b60 con 0x7f145c102120 2024-03-28T02:08:49.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.574+0000 7f1452ffd640 1 -- 172.21.15.106:0/1390007671 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14580049e0 con 0x7f145c102120 2024-03-28T02:08:49.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.574+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f145c1a20b0 con 0x7f145c102120 2024-03-28T02:08:49.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.574+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145c0fd620 con 0x7f145c102120 2024-03-28T02:08:49.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.576+0000 7f1452ffd640 1 -- 172.21.15.106:0/1390007671 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f14580079b0 con 0x7f145c102120 2024-03-28T02:08:49.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.576+0000 7f1452ffd640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f14480a2800 0x7f14480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:49.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.576+0000 7f14612b6640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f14480a2800 0x7f14480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:49.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.576+0000 7f1452ffd640 1 -- 172.21.15.106:0/1390007671 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f14580ce9e0 con 0x7f145c102120 2024-03-28T02:08:49.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.577+0000 7f14612b6640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f14480a2800 0x7f14480a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f145c101800 tx=0x7f144c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:49.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.580+0000 7f1452ffd640 1 -- 172.21.15.106:0/1390007671 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1458014030 con 0x7f145c102120 2024-03-28T02:08:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:49 smithi153 ceph-mon[36228]: pgmap v285: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:49 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3582471848' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T02:08:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:49 smithi153 ceph-mon[36228]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T02:08:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:49 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3675300232' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T02:08:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:49 smithi153 ceph-mon[36228]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T02:08:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:49 smithi153 ceph-mon[36228]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T02:08:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:49 smithi153 ceph-mon[36228]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T02:08:49.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:49.765+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f145c02d070 con 0x7f145c102120 2024-03-28T02:08:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:49 smithi106 ceph-mon[31965]: pgmap v285: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:49 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3582471848' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T02:08:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:49 smithi106 ceph-mon[31965]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T02:08:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:49 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3675300232' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T02:08:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:49 smithi106 ceph-mon[31965]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T02:08:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:49 smithi106 ceph-mon[31965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T02:08:49.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:49 smithi106 ceph-mon[31965]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T02:08:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:49 smithi178 ceph-mon[36521]: pgmap v285: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:49 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3582471848' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T02:08:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:49 smithi178 ceph-mon[36521]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T02:08:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:49 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3675300232' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T02:08:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:49 smithi178 ceph-mon[36521]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T02:08:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:49 smithi178 ceph-mon[36521]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T02:08:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:49 smithi178 ceph-mon[36521]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T02:08:50.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.241+0000 7f1452ffd640 1 -- 172.21.15.106:0/1390007671 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7f145809bbd0 con 0x7f145c102120 2024-03-28T02:08:50.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f14480a2800 msgr2=0x7f14480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:50.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f14480a2800 0x7f14480a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f145c101800 tx=0x7f144c008040 comp rx=0 tx=0).stop 2024-03-28T02:08:50.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c102120 msgr2=0x7f145c101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:50.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c102120 0x7f145c101a90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f145800c890 tx=0x7f145800cd50 comp rx=0 tx=0).stop 2024-03-28T02:08:50.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 shutdown_connections 2024-03-28T02:08:50.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f14480a2800 0x7f14480a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:50.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f145c107380 0x7f145c100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:50.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f145c104a50 0x7f145c1000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:50.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 --2- 172.21.15.106:0/1390007671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f145c102120 0x7f145c101a90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:50.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 >> 172.21.15.106:0/1390007671 conn(0x7f145c0fbb30 msgr2=0x7f145c1029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:50.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 shutdown_connections 2024-03-28T02:08:50.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.244+0000 7f1463d1b640 1 -- 172.21.15.106:0/1390007671 wait complete. 2024-03-28T02:08:50.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T02:08:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.364+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2012732486 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 msgr2=0x7f4cf4100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.364+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2012732486 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 0x7f4cf4100a60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009900 tx=0x7f4ce402bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.364+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2012732486 shutdown_connections 2024-03-28T02:08:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.364+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2012732486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf41021f0 0x7f4cf4108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.364+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2012732486 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cf4101830 0x7f4cf4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.364+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2012732486 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 0x7f4cf4100a60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.364+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2012732486 >> 172.21.15.106:0/2012732486 conn(0x7f4cf40fbbb0 msgr2=0x7f4cf40fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.364+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2012732486 shutdown_connections 2024-03-28T02:08:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.364+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2012732486 wait complete. 2024-03-28T02:08:50.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf8af1640 1 Processor -- start 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf8af1640 1 -- start start 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf8af1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 0x7f4cf41979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf8af1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf4101830 0x7f4cf4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf8af1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cf41021f0 0x7f4cf419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf8af1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf410d370 con 0x7f4cf4100640 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf8af1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf410d1f0 con 0x7f4cf4101830 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf8af1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf410d4f0 con 0x7f4cf41021f0 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 0x7f4cf41979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 0x7f4cf41979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53132/0 (socket says 172.21.15.106:53132) 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf259c640 1 -- 172.21.15.106:0/2764247286 learned_addr learned my addr 172.21.15.106:0/2764247286 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.365+0000 7f4cf2d9d640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cf41021f0 0x7f4cf419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf1d9b640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf4101830 0x7f4cf4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf259c640 1 -- 172.21.15.106:0/2764247286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cf41021f0 msgr2=0x7f4cf419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf259c640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cf41021f0 0x7f4cf419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf259c640 1 -- 172.21.15.106:0/2764247286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf4101830 msgr2=0x7f4cf4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf259c640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf4101830 0x7f4cf4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf259c640 1 -- 172.21.15.106:0/2764247286 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf419f5c0 con 0x7f4cf4100640 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf1d9b640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf4101830 0x7f4cf4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf259c640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 0x7f4cf41979a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4005ab0 tx=0x7f4ce4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4ce37fe640 1 -- 172.21.15.106:0/2764247286 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4041070 con 0x7f4cf4100640 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf419f850 con 0x7f4cf4100640 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4ce37fe640 1 -- 172.21.15.106:0/2764247286 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ce40054e0 con 0x7f4cf4100640 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4ce37fe640 1 -- 172.21.15.106:0/2764247286 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4034440 con 0x7f4cf4100640 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.366+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf419fd90 con 0x7f4cf4100640 2024-03-28T02:08:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.367+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf4107960 con 0x7f4cf4100640 2024-03-28T02:08:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.368+0000 7f4ce37fe640 1 -- 172.21.15.106:0/2764247286 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4ce4044070 con 0x7f4cf4100640 2024-03-28T02:08:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.368+0000 7f4ce37fe640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4cdc0a2800 0x7f4cdc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.369+0000 7f4cf1d9b640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4cdc0a2800 0x7f4cdc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.369+0000 7f4ce37fe640 1 -- 172.21.15.106:0/2764247286 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f4ce40f0710 con 0x7f4cf4100640 2024-03-28T02:08:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.369+0000 7f4cf1d9b640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4cdc0a2800 0x7f4cdc0a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80058c0 tx=0x7f4ce8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:50.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.373+0000 7f4ce37fe640 1 -- 172.21.15.106:0/2764247286 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ce40bd780 con 0x7f4cf4100640 2024-03-28T02:08:50.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:50.551+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4cf402d070 con 0x7f4cf4100640 2024-03-28T02:08:50.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:50 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3675300232' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T02:08:50.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:50 smithi106 ceph-mon[31965]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T02:08:50.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:50 smithi106 ceph-mon[31965]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T02:08:50.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:50 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1390007671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T02:08:50.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:50 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1390007671' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T02:08:50.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:50 smithi106 ceph-mon[31965]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T02:08:50.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:50 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3675300232' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T02:08:50.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:50 smithi153 ceph-mon[36228]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T02:08:50.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:50 smithi153 ceph-mon[36228]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T02:08:50.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:50 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1390007671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T02:08:50.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:50 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1390007671' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T02:08:50.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:50 smithi153 ceph-mon[36228]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T02:08:50.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:50 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3675300232' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T02:08:50.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:50 smithi178 ceph-mon[36521]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T02:08:50.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:50 smithi178 ceph-mon[36521]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T02:08:50.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:50 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1390007671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T02:08:50.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:50 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1390007671' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T02:08:50.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:50 smithi178 ceph-mon[36521]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T02:08:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:51 smithi153 ceph-mon[36228]: pgmap v289: 97 pgs: 16 creating+peering, 4 active+clean, 77 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:51 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] up:active 2024-03-28T02:08:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:51 smithi153 ceph-mon[36228]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:08:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:51 smithi153 ceph-mon[36228]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:08:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:51 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2764247286' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T02:08:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:51 smithi153 ceph-mon[36228]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T02:08:51.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:51 smithi153 ceph-mon[36228]: Cluster is now healthy 2024-03-28T02:08:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:51 smithi106 ceph-mon[31965]: pgmap v289: 97 pgs: 16 creating+peering, 4 active+clean, 77 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:51 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] up:active 2024-03-28T02:08:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:51 smithi106 ceph-mon[31965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:08:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:51 smithi106 ceph-mon[31965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:08:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:51 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2764247286' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T02:08:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:51 smithi106 ceph-mon[31965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T02:08:51.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:51 smithi106 ceph-mon[31965]: Cluster is now healthy 2024-03-28T02:08:51.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:51 smithi178 ceph-mon[36521]: pgmap v289: 97 pgs: 16 creating+peering, 4 active+clean, 77 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:08:51.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:51 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] up:active 2024-03-28T02:08:51.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:51 smithi178 ceph-mon[36521]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:08:51.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:51 smithi178 ceph-mon[36521]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:08:51.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:51 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2764247286' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T02:08:51.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:51 smithi178 ceph-mon[36521]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T02:08:51.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:51 smithi178 ceph-mon[36521]: Cluster is now healthy 2024-03-28T02:08:52.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.468+0000 7f4ce37fe640 1 -- 172.21.15.106:0/2764247286 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7f4ce40bd080 con 0x7f4cf4100640 2024-03-28T02:08:52.462 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'cephfs_data_ec' created 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4cdc0a2800 msgr2=0x7f4cdc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4cdc0a2800 0x7f4cdc0a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80058c0 tx=0x7f4ce8005850 comp rx=0 tx=0).stop 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 msgr2=0x7f4cf41979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 0x7f4cf41979a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4005ab0 tx=0x7f4ce4005dc0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 shutdown_connections 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cf41021f0 0x7f4cf419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4cdc0a2800 0x7f4cdc0a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf4101830 0x7f4cf4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 --2- 172.21.15.106:0/2764247286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4100640 0x7f4cf41979a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 >> 172.21.15.106:0/2764247286 conn(0x7f4cf40fbbb0 msgr2=0x7f4cf4104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.471+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 shutdown_connections 2024-03-28T02:08:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.472+0000 7f4cf8af1640 1 -- 172.21.15.106:0/2764247286 wait complete. 2024-03-28T02:08:52.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-28T02:08:52.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.596+0000 7fb870a44640 1 -- 172.21.15.106:0/3963567416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 msgr2=0x7fb86c101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:52.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.596+0000 7fb870a44640 1 --2- 172.21.15.106:0/3963567416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 0x7fb86c101af0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009900 tx=0x7fb85c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:52.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 -- 172.21.15.106:0/3963567416 shutdown_connections 2024-03-28T02:08:52.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 --2- 172.21.15.106:0/3963567416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb86c103280 0x7fb86c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 --2- 172.21.15.106:0/3963567416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb86c1028c0 0x7fb86c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 --2- 172.21.15.106:0/3963567416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 0x7fb86c101af0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 -- 172.21.15.106:0/3963567416 >> 172.21.15.106:0/3963567416 conn(0x7fb86c0fcc40 msgr2=0x7fb86c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 -- 172.21.15.106:0/3963567416 shutdown_connections 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 -- 172.21.15.106:0/3963567416 wait complete. 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 Processor -- start 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 -- start start 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 0x7fb86c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.597+0000 7fb870a44640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb86c1028c0 0x7fb86c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb870a44640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb86c103280 0x7fb86c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb870a44640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c10d320 con 0x7fb86c1016d0 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb870a44640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c10d1a0 con 0x7fb86c1028c0 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb870a44640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c10d4a0 con 0x7fb86c103280 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86a59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 0x7fb86c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86a59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 0x7fb86c199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53142/0 (socket says 172.21.15.106:53142) 2024-03-28T02:08:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86a59c640 1 -- 172.21.15.106:0/833526745 learned_addr learned my addr 172.21.15.106:0/833526745 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb869d9b640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb86c1028c0 0x7fb86c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86ad9d640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb86c103280 0x7fb86c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86a59c640 1 -- 172.21.15.106:0/833526745 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb86c103280 msgr2=0x7fb86c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86a59c640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb86c103280 0x7fb86c1a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86a59c640 1 -- 172.21.15.106:0/833526745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb86c1028c0 msgr2=0x7fb86c19a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86a59c640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb86c1028c0 0x7fb86c19a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86a59c640 1 -- 172.21.15.106:0/833526745 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c1a1840 con 0x7fb86c1016d0 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb86a59c640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 0x7fb86c199c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb85c000c00 tx=0x7fb85c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb85b7fe640 1 -- 172.21.15.106:0/833526745 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c041070 con 0x7fb86c1016d0 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb85b7fe640 1 -- 172.21.15.106:0/833526745 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb85c0044d0 con 0x7fb86c1016d0 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c1a1ad0 con 0x7fb86c1016d0 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb85b7fe640 1 -- 172.21.15.106:0/833526745 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c034440 con 0x7fb86c1016d0 2024-03-28T02:08:52.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.598+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c1a1fc0 con 0x7fb86c1016d0 2024-03-28T02:08:52.595 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.599+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c1a2190 con 0x7fb86c1016d0 2024-03-28T02:08:52.595 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.600+0000 7fb85b7fe640 1 -- 172.21.15.106:0/833526745 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb85c044070 con 0x7fb86c1016d0 2024-03-28T02:08:52.595 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.601+0000 7fb85b7fe640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb8540a2800 0x7fb8540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:52.596 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.601+0000 7fb869d9b640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb8540a2800 0x7fb8540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:52.596 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.601+0000 7fb85b7fe640 1 -- 172.21.15.106:0/833526745 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fb85c0f0ad0 con 0x7fb86c1016d0 2024-03-28T02:08:52.596 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.602+0000 7fb869d9b640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb8540a2800 0x7fb8540a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb860009bd0 tx=0x7fb860005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:52.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.605+0000 7fb85b7fe640 1 -- 172.21.15.106:0/833526745 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb85c0bd950 con 0x7fb86c1016d0 2024-03-28T02:08:52.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:52 smithi153 ceph-mon[36228]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T02:08:52.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:52.781+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fb86c1a2360 con 0x7fb86c1016d0 2024-03-28T02:08:52.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:52 smithi106 ceph-mon[31965]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T02:08:52.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:52 smithi178 ceph-mon[36521]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T02:08:53.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.470+0000 7fb85b7fe640 1 -- 172.21.15.106:0/833526745 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb85c0bd250 con 0x7fb86c1016d0 2024-03-28T02:08:53.463 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 4 allow_ec_overwrites to true 2024-03-28T02:08:53.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb8540a2800 msgr2=0x7fb8540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb8540a2800 0x7fb8540a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb860009bd0 tx=0x7fb860005f90 comp rx=0 tx=0).stop 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 msgr2=0x7fb86c199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 0x7fb86c199c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb85c000c00 tx=0x7fb85c005d50 comp rx=0 tx=0).stop 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 shutdown_connections 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb86c103280 0x7fb86c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb8540a2800 0x7fb8540a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb86c1028c0 0x7fb86c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 --2- 172.21.15.106:0/833526745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb86c1016d0 0x7fb86c199c30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.473+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 >> 172.21.15.106:0/833526745 conn(0x7fb86c0fcc40 msgr2=0x7fb86c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.474+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 shutdown_connections 2024-03-28T02:08:53.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.474+0000 7fb870a44640 1 -- 172.21.15.106:0/833526745 wait complete. 2024-03-28T02:08:53.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-28T02:08:53.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.603+0000 7fb18e46e640 1 -- 172.21.15.106:0/2844237671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 msgr2=0x7fb1881089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.603+0000 7fb18e46e640 1 --2- 172.21.15.106:0/2844237671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 0x7fb1881089f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb180009ca0 tx=0x7fb18002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.603+0000 7fb18e46e640 1 -- 172.21.15.106:0/2844237671 shutdown_connections 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.603+0000 7fb18e46e640 1 --2- 172.21.15.106:0/2844237671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 0x7fb1881089f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.603+0000 7fb18e46e640 1 --2- 172.21.15.106:0/2844237671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1881017a0 0x7fb188101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.603+0000 7fb18e46e640 1 --2- 172.21.15.106:0/2844237671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1881005b0 0x7fb1881009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.603+0000 7fb18e46e640 1 -- 172.21.15.106:0/2844237671 >> 172.21.15.106:0/2844237671 conn(0x7fb1880fbb40 msgr2=0x7fb1880fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.603+0000 7fb18e46e640 1 -- 172.21.15.106:0/2844237671 shutdown_connections 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.603+0000 7fb18e46e640 1 -- 172.21.15.106:0/2844237671 wait complete. 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18e46e640 1 Processor -- start 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18e46e640 1 -- start start 2024-03-28T02:08:53.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18e46e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1881005b0 0x7fb188197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18e46e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1881017a0 0x7fb188197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18e46e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 0x7fb18819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18e46e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18810c2a0 con 0x7fb188102160 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18e46e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18810c120 con 0x7fb1881017a0 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18e46e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18810c420 con 0x7fb1881005b0 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18ca0b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 0x7fb18819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18ca0b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 0x7fb18819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53156/0 (socket says 172.21.15.106:53156) 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18ca0b640 1 -- 172.21.15.106:0/3221916492 learned_addr learned my addr 172.21.15.106:0/3221916492 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb1877fe640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1881017a0 0x7fb188197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb187fff640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1881005b0 0x7fb188197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18ca0b640 1 -- 172.21.15.106:0/3221916492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1881005b0 msgr2=0x7fb188197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18ca0b640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1881005b0 0x7fb188197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18ca0b640 1 -- 172.21.15.106:0/3221916492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1881017a0 msgr2=0x7fb188197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:53.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18ca0b640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1881017a0 0x7fb188197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:53.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18ca0b640 1 -- 172.21.15.106:0/3221916492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18819f630 con 0x7fb188102160 2024-03-28T02:08:53.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.604+0000 7fb18ca0b640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 0x7fb18819efc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb18002c700 tx=0x7fb180032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:53.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.605+0000 7fb1857fa640 1 -- 172.21.15.106:0/3221916492 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180039040 con 0x7fb188102160 2024-03-28T02:08:53.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.605+0000 7fb1857fa640 1 -- 172.21.15.106:0/3221916492 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb180044070 con 0x7fb188102160 2024-03-28T02:08:53.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.605+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18819f8c0 con 0x7fb188102160 2024-03-28T02:08:53.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.605+0000 7fb1857fa640 1 -- 172.21.15.106:0/3221916492 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1800042a0 con 0x7fb188102160 2024-03-28T02:08:53.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.605+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18819fdb0 con 0x7fb188102160 2024-03-28T02:08:53.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.606+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1881a00d0 con 0x7fb188102160 2024-03-28T02:08:53.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.607+0000 7fb1857fa640 1 -- 172.21.15.106:0/3221916492 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb180003970 con 0x7fb188102160 2024-03-28T02:08:53.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.607+0000 7fb1857fa640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1700a2800 0x7fb1700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:53.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.607+0000 7fb187fff640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1700a2800 0x7fb1700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:53.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.607+0000 7fb1857fa640 1 -- 172.21.15.106:0/3221916492 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fb1800f1190 con 0x7fb188102160 2024-03-28T02:08:53.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.608+0000 7fb187fff640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1700a2800 0x7fb1700a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb188101600 tx=0x7fb178008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:53.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.612+0000 7fb1857fa640 1 -- 172.21.15.106:0/3221916492 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb1800be0a0 con 0x7fb188102160 2024-03-28T02:08:53.733 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:08:53 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b[67179]: 2024-03-28T02:08:53.416+0000 7f3a59697700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T02:08:53.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:53 smithi153 ceph-mon[36228]: pgmap v291: 97 pgs: 16 creating+peering, 73 active+clean, 8 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-28T02:08:53.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:53 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2764247286' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T02:08:53.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:53 smithi153 ceph-mon[36228]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T02:08:53.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:53 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/833526745' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T02:08:53.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:53.826+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fb1881a0750 con 0x7fb188102160 2024-03-28T02:08:53.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:53 smithi106 ceph-mon[31965]: pgmap v291: 97 pgs: 16 creating+peering, 73 active+clean, 8 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-28T02:08:53.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:53 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2764247286' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T02:08:53.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:53 smithi106 ceph-mon[31965]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T02:08:53.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:53 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/833526745' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T02:08:53.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:53 smithi178 ceph-mon[36521]: pgmap v291: 97 pgs: 16 creating+peering, 73 active+clean, 8 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-28T02:08:53.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:53 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2764247286' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T02:08:53.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:53 smithi178 ceph-mon[36521]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T02:08:53.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:53 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/833526745' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T02:08:54.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.484+0000 7fb1857fa640 1 -- 172.21.15.106:0/3221916492 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fb1800bd9a0 con 0x7fb188102160 2024-03-28T02:08:54.477 INFO:teuthology.orchestra.run.smithi106.stderr:added data pool 4 to fsmap 2024-03-28T02:08:54.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.486+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1700a2800 msgr2=0x7fb1700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.486+0000 7fb18e46e640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1700a2800 0x7fb1700a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb188101600 tx=0x7fb178008040 comp rx=0 tx=0).stop 2024-03-28T02:08:54.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.486+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 msgr2=0x7fb18819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.486+0000 7fb18e46e640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 0x7fb18819efc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb18002c700 tx=0x7fb180032f20 comp rx=0 tx=0).stop 2024-03-28T02:08:54.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.487+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 shutdown_connections 2024-03-28T02:08:54.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.487+0000 7fb18e46e640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb188102160 0x7fb18819efc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.487+0000 7fb18e46e640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb1700a2800 0x7fb1700a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.487+0000 7fb18e46e640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1881017a0 0x7fb188197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.487+0000 7fb18e46e640 1 --2- 172.21.15.106:0/3221916492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1881005b0 0x7fb188197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.487+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 >> 172.21.15.106:0/3221916492 conn(0x7fb1880fbb40 msgr2=0x7fb1880fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:54.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.487+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 shutdown_connections 2024-03-28T02:08:54.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.487+0000 7fb18e46e640 1 -- 172.21.15.106:0/3221916492 wait complete. 2024-03-28T02:08:54.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:54.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.607+0000 7f2580663640 1 -- 172.21.15.106:0/3396352902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ff310 msgr2=0x7f25780ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.607+0000 7f2580663640 1 --2- 172.21.15.106:0/3396352902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ff310 0x7f25780ff730 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2568009900 tx=0x7f256802bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.607+0000 7f2580663640 1 -- 172.21.15.106:0/3396352902 shutdown_connections 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.607+0000 7f2580663640 1 --2- 172.21.15.106:0/3396352902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2578107390 0x7f25781097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.607+0000 7f2580663640 1 --2- 172.21.15.106:0/3396352902 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25780ffd00 0x7f2578106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.607+0000 7f2580663640 1 --2- 172.21.15.106:0/3396352902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ff310 0x7f25780ff730 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.607+0000 7f2580663640 1 -- 172.21.15.106:0/3396352902 >> 172.21.15.106:0/3396352902 conn(0x7f25780faea0 msgr2=0x7f25780fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.607+0000 7f2580663640 1 -- 172.21.15.106:0/3396352902 shutdown_connections 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.607+0000 7f2580663640 1 -- 172.21.15.106:0/3396352902 wait complete. 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f2580663640 1 Processor -- start 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f2580663640 1 -- start start 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f2580663640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25780ff310 0x7f25781027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f2580663640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ffd00 0x7f2578100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:54.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f2580663640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2578107390 0x7f2578101320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f2580663640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257810d360 con 0x7f25780ffd00 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f2580663640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257810d1e0 con 0x7f25780ff310 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f2580663640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257810d4e0 con 0x7f2578107390 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257dbfe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ffd00 0x7f2578100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257dbfe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ffd00 0x7f2578100de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53170/0 (socket says 172.21.15.106:53170) 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257dbfe640 1 -- 172.21.15.106:0/1153048719 learned_addr learned my addr 172.21.15.106:0/1153048719 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257e3ff640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25780ff310 0x7f25781027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257dbfe640 1 -- 172.21.15.106:0/1153048719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2578107390 msgr2=0x7f2578101320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257dbfe640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2578107390 0x7f2578101320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257dbfe640 1 -- 172.21.15.106:0/1153048719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25780ff310 msgr2=0x7f25781027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257dbfe640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25780ff310 0x7f25781027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257dbfe640 1 -- 172.21.15.106:0/1153048719 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2578101bb0 con 0x7f25780ffd00 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.608+0000 7f257dbfe640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ffd00 0x7f2578100de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f257400d770 tx=0x7f257400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.609+0000 7f256f7fe640 1 -- 172.21.15.106:0/1153048719 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574017070 con 0x7f25780ffd00 2024-03-28T02:08:54.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.609+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25781a1b20 con 0x7f25780ffd00 2024-03-28T02:08:54.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.609+0000 7f256f7fe640 1 -- 172.21.15.106:0/1153048719 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2574005500 con 0x7f25780ffd00 2024-03-28T02:08:54.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.609+0000 7f256f7fe640 1 -- 172.21.15.106:0/1153048719 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574004e00 con 0x7f25780ffd00 2024-03-28T02:08:54.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.609+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25781a2020 con 0x7f25780ffd00 2024-03-28T02:08:54.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.610+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2578105c60 con 0x7f25780ffd00 2024-03-28T02:08:54.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.611+0000 7f256f7fe640 1 -- 172.21.15.106:0/1153048719 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2574004fa0 con 0x7f25780ffd00 2024-03-28T02:08:54.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.611+0000 7f256f7fe640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f25640a2800 0x7f25640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:54.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.611+0000 7f257e3ff640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f25640a2800 0x7f25640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:54.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.611+0000 7f256f7fe640 1 -- 172.21.15.106:0/1153048719 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f25740ceae0 con 0x7f25780ffd00 2024-03-28T02:08:54.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.612+0000 7f257e3ff640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f25640a2800 0x7f25640a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f256802c7b0 tx=0x7f2568005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:54.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.615+0000 7f256f7fe640 1 -- 172.21.15.106:0/1153048719 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f257409b940 con 0x7f25780ffd00 2024-03-28T02:08:54.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:54 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/833526745' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T02:08:54.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:54 smithi153 ceph-mon[36228]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T02:08:54.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:54 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3221916492' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T02:08:54.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.829+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f257802d2c0 con 0x7f25780ffd00 2024-03-28T02:08:54.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.830+0000 7f256f7fe640 1 -- 172.21.15.106:0/1153048719 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11339 (secure 0 0 0) 0x7f257409b240 con 0x7f25780ffd00 2024-03-28T02:08:54.823 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:54.823 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:08:53.828305+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24509,"qdb_cluster":[24509]},"id":1}]} 2024-03-28T02:08:54.823 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 17 2024-03-28T02:08:54.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.831+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f25640a2800 msgr2=0x7f25640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.831+0000 7f2580663640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f25640a2800 0x7f25640a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f256802c7b0 tx=0x7f2568005fd0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.831+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ffd00 msgr2=0x7f2578100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.831+0000 7f2580663640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ffd00 0x7f2578100de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f257400d770 tx=0x7f257400dc30 comp rx=0 tx=0).stop 2024-03-28T02:08:54.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.832+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 shutdown_connections 2024-03-28T02:08:54.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.832+0000 7f2580663640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2578107390 0x7f2578101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.832+0000 7f2580663640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25780ffd00 0x7f2578100de0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.832+0000 7f2580663640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f25640a2800 0x7f25640a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.832+0000 7f2580663640 1 --2- 172.21.15.106:0/1153048719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25780ff310 0x7f25781027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.832+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 >> 172.21.15.106:0/1153048719 conn(0x7f25780faea0 msgr2=0x7f2578102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:54.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.832+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 shutdown_connections 2024-03-28T02:08:54.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.832+0000 7f2580663640 1 -- 172.21.15.106:0/1153048719 wait complete. 2024-03-28T02:08:54.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:54 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/833526745' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T02:08:54.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:54 smithi106 ceph-mon[31965]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T02:08:54.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:54 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3221916492' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T02:08:54.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:08:54.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:54 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/833526745' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T02:08:54.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:54 smithi178 ceph-mon[36521]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T02:08:54.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:54 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3221916492' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T02:08:54.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.943+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1326488690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 msgr2=0x7f66c8101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.943+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1326488690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 0x7f66c8101ae0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009900 tx=0x7f66c002bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:54.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1326488690 shutdown_connections 2024-03-28T02:08:54.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1326488690 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66c8103270 0x7f66c8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1326488690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c81028b0 0x7f66c8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1326488690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 0x7f66c8101ae0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1326488690 >> 172.21.15.106:0/1326488690 conn(0x7f66c80fcc50 msgr2=0x7f66c80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1326488690 shutdown_connections 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1326488690 wait complete. 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 Processor -- start 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 -- start start 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 0x7f66c81979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66c81028b0 0x7f66c8197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c8103270 0x7f66c819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c810d340 con 0x7f66c81016c0 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c810d1c0 con 0x7f66c8103270 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66cd6ac640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c810d4c0 con 0x7f66c81028b0 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.944+0000 7f66c6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 0x7f66c81979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:54.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 0x7f66c81979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53198/0 (socket says 172.21.15.106:53198) 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c6ffd640 1 -- 172.21.15.106:0/1516725398 learned_addr learned my addr 172.21.15.106:0/1516725398 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c77fe640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c8103270 0x7f66c819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c67fc640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66c81028b0 0x7f66c8197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c6ffd640 1 -- 172.21.15.106:0/1516725398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66c81028b0 msgr2=0x7f66c8197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c6ffd640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66c81028b0 0x7f66c8197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c6ffd640 1 -- 172.21.15.106:0/1516725398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c8103270 msgr2=0x7f66c819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c6ffd640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c8103270 0x7f66c819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c6ffd640 1 -- 172.21.15.106:0/1516725398 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c819f5d0 con 0x7f66c81016c0 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c67fc640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66c81028b0 0x7f66c8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66c6ffd640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 0x7f66c81979b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f66c0005ab0 tx=0x7f66c0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66bffff640 1 -- 172.21.15.106:0/1516725398 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0041070 con 0x7f66c81016c0 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66bffff640 1 -- 172.21.15.106:0/1516725398 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66c00054e0 con 0x7f66c81016c0 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c819f860 con 0x7f66c81016c0 2024-03-28T02:08:54.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66bffff640 1 -- 172.21.15.106:0/1516725398 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0034440 con 0x7f66c81016c0 2024-03-28T02:08:54.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.945+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c819fd50 con 0x7f66c81016c0 2024-03-28T02:08:54.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.946+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c81089e0 con 0x7f66c81016c0 2024-03-28T02:08:54.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.947+0000 7f66bffff640 1 -- 172.21.15.106:0/1516725398 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f66c0044070 con 0x7f66c81016c0 2024-03-28T02:08:54.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.948+0000 7f66bffff640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66b40a2800 0x7f66b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:54.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.948+0000 7f66c67fc640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66b40a2800 0x7f66b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:54.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.948+0000 7f66bffff640 1 -- 172.21.15.106:0/1516725398 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f66c00f0910 con 0x7f66c81016c0 2024-03-28T02:08:54.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.948+0000 7f66c67fc640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66b40a2800 0x7f66b40a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f66c8198ec0 tx=0x7f66b00093e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:54.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:54.952+0000 7f66bffff640 1 -- 172.21.15.106:0/1516725398 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f66c00bd770 con 0x7f66c81016c0 2024-03-28T02:08:55.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.128+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66c802d2b0 con 0x7f66c81016c0 2024-03-28T02:08:55.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.130+0000 7f66bffff640 1 -- 172.21.15.106:0/1516725398 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7f66c00bd070 con 0x7f66c81016c0 2024-03-28T02:08:55.123 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:08:55.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66b40a2800 msgr2=0x7f66b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66b40a2800 0x7f66b40a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f66c8198ec0 tx=0x7f66b00093e0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 msgr2=0x7f66c81979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 0x7f66c81979b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f66c0005ab0 tx=0x7f66c0005dc0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 shutdown_connections 2024-03-28T02:08:55.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66b40a2800 0x7f66b40a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c8103270 0x7f66c819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66c81028b0 0x7f66c8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 --2- 172.21.15.106:0/1516725398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81016c0 0x7f66c81979b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 >> 172.21.15.106:0/1516725398 conn(0x7f66c80fcc50 msgr2=0x7f66c8105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:55.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 shutdown_connections 2024-03-28T02:08:55.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.132+0000 7f66cd6ac640 1 -- 172.21.15.106:0/1516725398 wait complete. 2024-03-28T02:08:55.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:08:55.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 -- 172.21.15.106:0/3059619225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 msgr2=0x7f5184100190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 --2- 172.21.15.106:0/3059619225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 0x7f5184100190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5174009900 tx=0x7f517402bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 -- 172.21.15.106:0/3059619225 shutdown_connections 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 --2- 172.21.15.106:0/3059619225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5184101920 0x7f51841081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 --2- 172.21.15.106:0/3059619225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5184100f60 0x7f51841013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 --2- 172.21.15.106:0/3059619225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 0x7f5184100190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 -- 172.21.15.106:0/3059619225 >> 172.21.15.106:0/3059619225 conn(0x7f51840fb2e0 msgr2=0x7f51840fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 -- 172.21.15.106:0/3059619225 shutdown_connections 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 -- 172.21.15.106:0/3059619225 wait complete. 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.245+0000 7f518ba9a640 1 Processor -- start 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f518ba9a640 1 -- start start 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f518ba9a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 0x7f5184195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f518ba9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5184100f60 0x7f5184195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f518ba9a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5184101920 0x7f518419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f518ba9a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518410b180 con 0x7f51840ffd70 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f518ba9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518410b000 con 0x7f5184100f60 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f518ba9a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518410b300 con 0x7f5184101920 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189836640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 0x7f5184195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189836640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 0x7f5184195840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53214/0 (socket says 172.21.15.106:53214) 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189836640 1 -- 172.21.15.106:0/23634361 learned_addr learned my addr 172.21.15.106:0/23634361 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189035640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5184100f60 0x7f5184195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189836640 1 -- 172.21.15.106:0/23634361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5184101920 msgr2=0x7f518419cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189836640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5184101920 0x7f518419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189836640 1 -- 172.21.15.106:0/23634361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5184100f60 msgr2=0x7f5184195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189836640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5184100f60 0x7f5184195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189836640 1 -- 172.21.15.106:0/23634361 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518419d460 con 0x7f51840ffd70 2024-03-28T02:08:55.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f5189836640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 0x7f5184195840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5174000c00 tx=0x7f5174005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:55.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f517affd640 1 -- 172.21.15.106:0/23634361 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174041070 con 0x7f51840ffd70 2024-03-28T02:08:55.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f517affd640 1 -- 172.21.15.106:0/23634361 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51740044d0 con 0x7f51840ffd70 2024-03-28T02:08:55.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518419d6f0 con 0x7f51840ffd70 2024-03-28T02:08:55.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f517affd640 1 -- 172.21.15.106:0/23634361 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174034440 con 0x7f51840ffd70 2024-03-28T02:08:55.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.246+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518419dc30 con 0x7f51840ffd70 2024-03-28T02:08:55.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.247+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5184107160 con 0x7f51840ffd70 2024-03-28T02:08:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.248+0000 7f517affd640 1 -- 172.21.15.106:0/23634361 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5174044070 con 0x7f51840ffd70 2024-03-28T02:08:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.249+0000 7f517affd640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51700a4a90 0x7f51700a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.249+0000 7f5189035640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51700a4a90 0x7f51700a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.249+0000 7f517affd640 1 -- 172.21.15.106:0/23634361 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f51740f07c0 con 0x7f51840ffd70 2024-03-28T02:08:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.249+0000 7f5189035640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51700a4a90 0x7f51700a6f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5184196d50 tx=0x7f5180010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:55.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.253+0000 7f517affd640 1 -- 172.21.15.106:0/23634361 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f51740bd620 con 0x7f51840ffd70 2024-03-28T02:08:55.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.429+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5184062340 con 0x7f51840ffd70 2024-03-28T02:08:55.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.431+0000 7f517affd640 1 -- 172.21.15.106:0/23634361 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7f51740bcf20 con 0x7f51840ffd70 2024-03-28T02:08:55.424 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:55.424 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51700a4a90 msgr2=0x7f51700a6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51700a4a90 0x7f51700a6f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5184196d50 tx=0x7f5180010040 comp rx=0 tx=0).stop 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 msgr2=0x7f5184195840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 0x7f5184195840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5174000c00 tx=0x7f5174005d50 comp rx=0 tx=0).stop 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 shutdown_connections 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5184101920 0x7f518419cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51700a4a90 0x7f51700a6f50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5184100f60 0x7f5184195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 --2- 172.21.15.106:0/23634361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51840ffd70 0x7f5184195840 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 >> 172.21.15.106:0/23634361 conn(0x7f51840fb2e0 msgr2=0x7f5184104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 shutdown_connections 2024-03-28T02:08:55.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.433+0000 7f518ba9a640 1 -- 172.21.15.106:0/23634361 wait complete. 2024-03-28T02:08:55.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:08:55.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.560+0000 7f580eaa1640 1 -- 172.21.15.106:0/491017785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 msgr2=0x7f58081089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.560+0000 7f580eaa1640 1 --2- 172.21.15.106:0/491017785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 0x7f58081089f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009900 tx=0x7f57f002bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:55.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 -- 172.21.15.106:0/491017785 shutdown_connections 2024-03-28T02:08:55.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 --2- 172.21.15.106:0/491017785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 0x7f58081089f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 --2- 172.21.15.106:0/491017785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58081017a0 0x7f5808101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 --2- 172.21.15.106:0/491017785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58081005b0 0x7f58081009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 -- 172.21.15.106:0/491017785 >> 172.21.15.106:0/491017785 conn(0x7f58080fbb40 msgr2=0x7f58080fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 -- 172.21.15.106:0/491017785 shutdown_connections 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 -- 172.21.15.106:0/491017785 wait complete. 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 Processor -- start 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 -- start start 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58081005b0 0x7f5808197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58081017a0 0x7f5808197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 0x7f580819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580810c210 con 0x7f5808102160 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580810c090 con 0x7f58081005b0 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580eaa1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580810c390 con 0x7f58081017a0 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580d03e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 0x7f580819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580d03e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 0x7f580819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53238/0 (socket says 172.21.15.106:53238) 2024-03-28T02:08:55.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580d03e640 1 -- 172.21.15.106:0/225023268 learned_addr learned my addr 172.21.15.106:0/225023268 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f580c83d640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58081005b0 0x7f5808197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.561+0000 7f5807fff640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58081017a0 0x7f5808197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f580d03e640 1 -- 172.21.15.106:0/225023268 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58081017a0 msgr2=0x7f5808197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f580d03e640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58081017a0 0x7f5808197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f580d03e640 1 -- 172.21.15.106:0/225023268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58081005b0 msgr2=0x7f5808197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f580d03e640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58081005b0 0x7f5808197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f580d03e640 1 -- 172.21.15.106:0/225023268 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580819f590 con 0x7f5808102160 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f580d03e640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 0x7f580819ef20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f57f002c7b0 tx=0x7f57f0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f5805ffb640 1 -- 172.21.15.106:0/225023268 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0041070 con 0x7f5808102160 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580819f820 con 0x7f5808102160 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f5805ffb640 1 -- 172.21.15.106:0/225023268 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57f00054e0 con 0x7f5808102160 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f5805ffb640 1 -- 172.21.15.106:0/225023268 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0034440 con 0x7f5808102160 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.562+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580819fd60 con 0x7f5808102160 2024-03-28T02:08:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.563+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580819ffc0 con 0x7f5808102160 2024-03-28T02:08:55.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.564+0000 7f5805ffb640 1 -- 172.21.15.106:0/225023268 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f57f0044070 con 0x7f5808102160 2024-03-28T02:08:55.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.565+0000 7f5805ffb640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f57f40a2800 0x7f57f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.565+0000 7f580c83d640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f57f40a2800 0x7f57f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.565+0000 7f5805ffb640 1 -- 172.21.15.106:0/225023268 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f57f00f0ca0 con 0x7f5808102160 2024-03-28T02:08:55.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.565+0000 7f580c83d640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f57f40a2800 0x7f57f40a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5808101600 tx=0x7f5800005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:55.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.569+0000 7f5805ffb640 1 -- 172.21.15.106:0/225023268 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f57f00bd4b0 con 0x7f5808102160 2024-03-28T02:08:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:55 smithi153 ceph-mon[36228]: pgmap v294: 129 pgs: 4 creating+peering, 99 active+clean, 26 unknown; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 10 op/s 2024-03-28T02:08:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:55 smithi153 ceph-mon[36228]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T02:08:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:55 smithi153 ceph-mon[36228]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T02:08:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:55 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3221916492' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T02:08:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:55 smithi153 ceph-mon[36228]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:08:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:55 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1153048719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:55 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1516725398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:55.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:55 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/23634361' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:55.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.781+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58081a0190 con 0x7f5808102160 2024-03-28T02:08:55.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.781+0000 7f5805ffb640 1 -- 172.21.15.106:0/225023268 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11339 (secure 0 0 0) 0x7f57f00c1550 con 0x7f5808102160 2024-03-28T02:08:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:08:53.828305+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24509,"qdb_cluster":[24509]},"id":1}]} 2024-03-28T02:08:55.775 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 17 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f57f40a2800 msgr2=0x7f57f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f57f40a2800 0x7f57f40a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5808101600 tx=0x7f5800005f90 comp rx=0 tx=0).stop 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 msgr2=0x7f580819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 0x7f580819ef20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f57f002c7b0 tx=0x7f57f0005dd0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 shutdown_connections 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5808102160 0x7f580819ef20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58081017a0 0x7f5808197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f57f40a2800 0x7f57f40a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 --2- 172.21.15.106:0/225023268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58081005b0 0x7f5808197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 >> 172.21.15.106:0/225023268 conn(0x7f58080fbb40 msgr2=0x7f58080fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:55.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 shutdown_connections 2024-03-28T02:08:55.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.783+0000 7f580eaa1640 1 -- 172.21.15.106:0/225023268 wait complete. 2024-03-28T02:08:55.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:55 smithi106 ceph-mon[31965]: pgmap v294: 129 pgs: 4 creating+peering, 99 active+clean, 26 unknown; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 10 op/s 2024-03-28T02:08:55.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:55 smithi106 ceph-mon[31965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T02:08:55.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:55 smithi106 ceph-mon[31965]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T02:08:55.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:55 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3221916492' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T02:08:55.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:55 smithi106 ceph-mon[31965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:08:55.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:55 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1153048719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:55.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:55 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1516725398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:55.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:55 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/23634361' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:55.788 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:08:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:55 smithi178 ceph-mon[36521]: pgmap v294: 129 pgs: 4 creating+peering, 99 active+clean, 26 unknown; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 10 op/s 2024-03-28T02:08:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:55 smithi178 ceph-mon[36521]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T02:08:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:55 smithi178 ceph-mon[36521]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T02:08:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:55 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3221916492' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T02:08:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:55 smithi178 ceph-mon[36521]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:08:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:55 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1153048719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:55 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1516725398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:55.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:55 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/23634361' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:55.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.897+0000 7f8c670ff640 1 -- 172.21.15.106:0/1638424419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60107420 msgr2=0x7f8c60109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.897+0000 7f8c670ff640 1 --2- 172.21.15.106:0/1638424419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60107420 0x7f8c60109810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8c58009ca0 tx=0x7f8c5802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.897+0000 7f8c670ff640 1 -- 172.21.15.106:0/1638424419 shutdown_connections 2024-03-28T02:08:55.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.897+0000 7f8c670ff640 1 --2- 172.21.15.106:0/1638424419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60107420 0x7f8c60109810 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.897+0000 7f8c670ff640 1 --2- 172.21.15.106:0/1638424419 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c60104af0 0x7f8c60106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.897+0000 7f8c670ff640 1 --2- 172.21.15.106:0/1638424419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c601021c0 0x7f8c601045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.897+0000 7f8c670ff640 1 -- 172.21.15.106:0/1638424419 >> 172.21.15.106:0/1638424419 conn(0x7f8c600fbb50 msgr2=0x7f8c600fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.897+0000 7f8c670ff640 1 -- 172.21.15.106:0/1638424419 shutdown_connections 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.897+0000 7f8c670ff640 1 -- 172.21.15.106:0/1638424419 wait complete. 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c670ff640 1 Processor -- start 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c670ff640 1 -- start start 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c670ff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c601021c0 0x7f8c60197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c670ff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60104af0 0x7f8c60197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c670ff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c60107420 0x7f8c6019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c670ff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6010d360 con 0x7f8c60104af0 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c670ff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6010d1e0 con 0x7f8c601021c0 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c670ff640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6010d4e0 con 0x7f8c60107420 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60104af0 0x7f8c60197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60104af0 0x7f8c60197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53262/0 (socket says 172.21.15.106:53262) 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5ffff640 1 -- 172.21.15.106:0/499673303 learned_addr learned my addr 172.21.15.106:0/499673303 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c64e9b640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c601021c0 0x7f8c60197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5ffff640 1 -- 172.21.15.106:0/499673303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c60107420 msgr2=0x7f8c6019eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5ffff640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c60107420 0x7f8c6019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5ffff640 1 -- 172.21.15.106:0/499673303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c601021c0 msgr2=0x7f8c60197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5ffff640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c601021c0 0x7f8c60197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5ffff640 1 -- 172.21.15.106:0/499673303 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6019f540 con 0x7f8c60104af0 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5ffff640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60104af0 0x7f8c60197e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c970 tx=0x7f8c4c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5dffb640 1 -- 172.21.15.106:0/499673303 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c014070 con 0x7f8c60104af0 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5dffb640 1 -- 172.21.15.106:0/499673303 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c4c004440 con 0x7f8c60104af0 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6019f830 con 0x7f8c60104af0 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.898+0000 7f8c5dffb640 1 -- 172.21.15.106:0/499673303 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c00f450 con 0x7f8c60104af0 2024-03-28T02:08:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.899+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6019fd80 con 0x7f8c60104af0 2024-03-28T02:08:55.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.900+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c600fd640 con 0x7f8c60104af0 2024-03-28T02:08:55.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.900+0000 7f8c5dffb640 1 -- 172.21.15.106:0/499673303 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8c4c017070 con 0x7f8c60104af0 2024-03-28T02:08:55.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.901+0000 7f8c5dffb640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8c500a2800 0x7f8c500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:55.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.901+0000 7f8c64e9b640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8c500a2800 0x7f8c500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:55.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.901+0000 7f8c5dffb640 1 -- 172.21.15.106:0/499673303 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8c4c0ceaf0 con 0x7f8c60104af0 2024-03-28T02:08:55.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.901+0000 7f8c64e9b640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8c500a2800 0x7f8c500a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009bd0 tx=0x7f8c54005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:55.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:55.905+0000 7f8c5dffb640 1 -- 172.21.15.106:0/499673303 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8c4c01e020 con 0x7f8c60104af0 2024-03-28T02:08:56.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.080+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c600622a0 con 0x7f8c60104af0 2024-03-28T02:08:56.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.082+0000 7f8c5dffb640 1 -- 172.21.15.106:0/499673303 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7f8c4c09b950 con 0x7f8c60104af0 2024-03-28T02:08:56.075 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:08:56.076 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:08:56.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8c500a2800 msgr2=0x7f8c500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8c500a2800 0x7f8c500a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009bd0 tx=0x7f8c54005f90 comp rx=0 tx=0).stop 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60104af0 msgr2=0x7f8c60197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60104af0 0x7f8c60197e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c970 tx=0x7f8c4c00ce30 comp rx=0 tx=0).stop 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 shutdown_connections 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c60107420 0x7f8c6019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c60104af0 0x7f8c60197e60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8c500a2800 0x7f8c500a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 --2- 172.21.15.106:0/499673303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c601021c0 0x7f8c60197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 >> 172.21.15.106:0/499673303 conn(0x7f8c600fbb50 msgr2=0x7f8c60102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 shutdown_connections 2024-03-28T02:08:56.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.084+0000 7f8c670ff640 1 -- 172.21.15.106:0/499673303 wait complete. 2024-03-28T02:08:56.089 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T02:08:48.782692+0000', 'modified': '2024-03-28T02:08:53.828305+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24509}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6839/505906484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 505906484}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 505906484}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24509, 'qdb_cluster': [24509]} 2024-03-28T02:08:56.089 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T02:08:56.089 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T02:08:56.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 -- 172.21.15.106:0/2364581867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef01005a0 msgr2=0x7f4ef01009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2364581867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef01005a0 0x7f4ef01009c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009900 tx=0x7f4ee002bc30 comp rx=0 tx=0).stop 2024-03-28T02:08:56.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 -- 172.21.15.106:0/2364581867 shutdown_connections 2024-03-28T02:08:56.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2364581867 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ef0102150 0x7f4ef01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2364581867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0101790 0x7f4ef0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2364581867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef01005a0 0x7f4ef01009c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 -- 172.21.15.106:0/2364581867 >> 172.21.15.106:0/2364581867 conn(0x7f4ef00fbb30 msgr2=0x7f4ef00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:56.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 -- 172.21.15.106:0/2364581867 shutdown_connections 2024-03-28T02:08:56.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 -- 172.21.15.106:0/2364581867 wait complete. 2024-03-28T02:08:56.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 Processor -- start 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 -- start start 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.198+0000 7f4ef5562640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ef01005a0 0x7f4ef0196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4ef5562640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0101790 0x7f4ef0196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4ef5562640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0102150 0x7f4ef019dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4ef5562640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef010c230 con 0x7f4ef0102150 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4ef5562640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef010c0b0 con 0x7f4ef0101790 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4ef5562640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef010c3b0 con 0x7f4ef01005a0 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eef7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0102150 0x7f4ef019dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4ee7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0101790 0x7f4ef0196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eef7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0102150 0x7f4ef019dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53272/0 (socket says 172.21.15.106:53272) 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eeeffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ef01005a0 0x7f4ef0196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eef7fe640 1 -- 172.21.15.106:0/2013129074 learned_addr learned my addr 172.21.15.106:0/2013129074 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eef7fe640 1 -- 172.21.15.106:0/2013129074 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ef01005a0 msgr2=0x7f4ef0196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eef7fe640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ef01005a0 0x7f4ef0196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eef7fe640 1 -- 172.21.15.106:0/2013129074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0101790 msgr2=0x7f4ef0196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eef7fe640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0101790 0x7f4ef0196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eef7fe640 1 -- 172.21.15.106:0/2013129074 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef019e530 con 0x7f4ef0102150 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0101790 0x7f4ef0196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eef7fe640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0102150 0x7f4ef019dec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800eb10 tx=0x7f4ee800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eecff9640 1 -- 172.21.15.106:0/2013129074 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8004b20 con 0x7f4ef0102150 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eecff9640 1 -- 172.21.15.106:0/2013129074 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ee8004cc0 con 0x7f4ef0102150 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4ef5562640 1 -- 172.21.15.106:0/2013129074 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef019e820 con 0x7f4ef0102150 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4eecff9640 1 -- 172.21.15.106:0/2013129074 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8010760 con 0x7f4ef0102150 2024-03-28T02:08:56.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.199+0000 7f4ef5562640 1 -- 172.21.15.106:0/2013129074 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef019ed70 con 0x7f4ef0102150 2024-03-28T02:08:56.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4eecff9640 1 -- 172.21.15.106:0/2013129074 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4ee8005020 con 0x7f4ef0102150 2024-03-28T02:08:56.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ef822d740 1 Processor -- start 2024-03-28T02:08:56.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ef822d740 1 -- start start 2024-03-28T02:08:56.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068bfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 0x555b068c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b068c7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ef822d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b06824c20 con 0x555b068bf740 2024-03-28T02:08:56.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ef822d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b067cda40 con 0x555b068c52a0 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ef822d740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b06820cc0 con 0x555b068c0130 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ee7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068bfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ee7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068bfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53288/0 (socket says 172.21.15.106:53288) 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ee7fff640 1 -- 172.21.15.106:0/4023889355 learned_addr learned my addr 172.21.15.106:0/4023889355 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4eecff9640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4edc0a2800 0x7f4edc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.202+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b068c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4eeeffd640 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 0x555b068c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4ee7fff640 1 -- 172.21.15.106:0/4023889355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 msgr2=0x555b068c4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 0x555b068c4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4ee7fff640 1 -- 172.21.15.106:0/4023889355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 msgr2=0x555b068c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b068c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4eef7fe640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4edc0a2800 0x7f4edc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4ee7fff640 1 -- 172.21.15.106:0/4023889355 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b067eff90 con 0x555b068bf740 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4eeeffd640 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 0x555b068c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4eecff9640 1 -- 172.21.15.106:0/2013129074 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4ee80cf380 con 0x7f4ef0102150 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068bfb60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8013a20 tx=0x7f4ed8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43f0f2e7f0288bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4ee57fa640 1 -- 172.21.15.106:0/4023889355 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8004030 con 0x555b068bf740 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4ee57fa640 1 -- 172.21.15.106:0/4023889355 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ed800fe60 con 0x555b068bf740 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.203+0000 7f4ee57fa640 1 -- 172.21.15.106:0/4023889355 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8003b00 con 0x555b068bf740 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 -- 172.21.15.106:0/4023889355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 msgr2=0x555b068bfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4eef7fe640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4edc0a2800 0x7f4edc0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8021000 tx=0x7f4ee806e810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068bfb60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8013a20 tx=0x7f4ed8009c00 comp rx=0 tx=0).stop 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 -- 172.21.15.106:0/4023889355 shutdown_connections 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b068c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 0x555b068c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4023889355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068bfb60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 -- 172.21.15.106:0/4023889355 >> 172.21.15.106:0/4023889355 conn(0x555b06887a60 msgr2=0x555b068bb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 -- 172.21.15.106:0/4023889355 shutdown_connections 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 -- 172.21.15.106:0/4023889355 wait complete. 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 -- start start 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 Processor -- start 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 0x555b068cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c0130 0x555b068cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c52a0 0x555b068d3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b068c81f0 con 0x555b068c52a0 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b067e0e90 con 0x555b068c0130 2024-03-28T02:08:56.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b068c8370 con 0x555b068bf740 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.204+0000 7f4ef822d740 10 client.0 _ll_register_callbacks cb 0x555b06538ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c52a0 0x555b068d3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c52a0 0x555b068d3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53296/0 (socket says 172.21.15.106:53296) 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 -- 172.21.15.106:0/4033060927 learned_addr learned my addr 172.21.15.106:0/4033060927 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4eeeffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c0130 0x555b068cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 0x555b068cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 -- 172.21.15.106:0/4033060927 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 msgr2=0x555b068cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 0x555b068cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 -- 172.21.15.106:0/4033060927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c0130 msgr2=0x555b068cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c0130 0x555b068cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 -- 172.21.15.106:0/4033060927 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b067ed9d0 con 0x555b068c52a0 2024-03-28T02:08:56.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4eeeffd640 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c0130 0x555b068cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c52a0 0x555b068d3370 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800f790 tx=0x7f4ed80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9964eaeefb669ef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ed67fc640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ed67fc640 1 -- 172.21.15.106:0/4033060927 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8022070 con 0x555b068c52a0 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ed67fc640 1 -- 172.21.15.106:0/4033060927 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4ed8022210 con 0x555b068c52a0 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ef822d740 1 -- 172.21.15.106:0/4033060927 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ed800fe60 con 0x555b068c52a0 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ed67fc640 1 -- 172.21.15.106:0/4033060927 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ed8010640 con 0x555b068c52a0 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ef822d740 10 client.14760 fetch_fsmap learned FSMap version 17 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ef822d740 1 -- 172.21.15.106:0/4033060927 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x555b065392d0 con 0x555b068c52a0 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.205+0000 7f4ed67fc640 1 -- 172.21.15.106:0/4033060927 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ed800fe60 con 0x555b068c52a0 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4ed67fc640 1 -- 172.21.15.106:0/4033060927 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f4ed801e070 con 0x555b068c52a0 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4ef822d740 10 client.14760 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4ef822d740 10 client.14760 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T02:08:56.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4ef822d740 20 client.14760 populate_metadata read hostname 'smithi106' 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x555b068d8290 0x555b068d7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4ef822d740 4 client.14760 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4ef822d740 1 -- 172.21.15.106:0/4033060927 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555b06832160 con 0x555b068d8290 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4eeeffd640 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x555b068d8290 0x555b068d7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4eeeffd640 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x555b068d8290 0x555b068d7530 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=717fb18d26c6366c server_cookie=d7d74ed11195294d in_seq=0 out_seq=0 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.206+0000 7f4ed67fc640 10 client.14760 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ed67fc640 1 -- 172.21.15.106:0/4033060927 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555b06832160 con 0x555b068d8290 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ed67fc640 10 client.14760 handle_command_reply: tid=0 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 client.14760 shutdown 2024-03-28T02:08:56.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 -- 172.21.15.106:0/4033060927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c52a0 msgr2=0x555b068d3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c52a0 0x555b068d3370 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800f790 tx=0x7f4ed80035f0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 -- 172.21.15.106:0/4033060927 shutdown_connections 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x555b068d8290 0x555b068d7530 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c52a0 0x555b068d3370 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c0130 0x555b068cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ed67fc640 0 client.14760 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 --2- 172.21.15.106:0/4033060927 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 0x555b068cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 -- 172.21.15.106:0/4033060927 >> 172.21.15.106:0/4033060927 conn(0x555b06887a60 msgr2=0x555b068ba2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 -- 172.21.15.106:0/4033060927 shutdown_connections 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 1 -- 172.21.15.106:0/4033060927 wait complete. 2024-03-28T02:08:56.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.207+0000 7f4ef822d740 20 client.14760 trim_cache size 0 max 16384 2024-03-28T02:08:56.223 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.223+0000 7f4ef822d740 1 Processor -- start 2024-03-28T02:08:56.223 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ef822d740 1 -- start start 2024-03-28T02:08:56.223 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068cba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.223 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 0x555b068d3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.223 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b068cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.223 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ef822d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b068d7ab0 con 0x555b068bf740 2024-03-28T02:08:56.223 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ef822d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b068d8110 con 0x555b068c52a0 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ef822d740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b068d8770 con 0x555b068c0130 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068cba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53310/0 (socket says 172.21.15.106:53310) 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 -- 172.21.15.106:0/1908677306 learned_addr learned my addr 172.21.15.106:0/1908677306 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/1908677306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b068cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 -- 172.21.15.106:0/1908677306 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 msgr2=0x555b068d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/1908677306 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 0x555b068d3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 -- 172.21.15.106:0/1908677306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 msgr2=0x555b068cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/1908677306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b068cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 -- 172.21.15.106:0/1908677306 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b066036b0 con 0x555b068bf740 2024-03-28T02:08:56.224 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/1908677306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068cba70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8000c00 tx=0x7f4ed80137e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4340d5d11e290cff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.224+0000 7f4ee4ff9640 1 -- 172.21.15.106:0/1908677306 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed800fdf0 con 0x555b068bf740 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ee4ff9640 1 -- 172.21.15.106:0/1908677306 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ed8020410 con 0x555b068bf740 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ee4ff9640 1 -- 172.21.15.106:0/1908677306 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8010b60 con 0x555b068bf740 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 -- 172.21.15.106:0/1908677306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 msgr2=0x555b068cba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 --2- 172.21.15.106:0/1908677306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068cba70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8000c00 tx=0x7f4ed80137e0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 -- 172.21.15.106:0/1908677306 shutdown_connections 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 --2- 172.21.15.106:0/1908677306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b068cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 --2- 172.21.15.106:0/1908677306 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068c0130 0x555b068d3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 --2- 172.21.15.106:0/1908677306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068bf740 0x555b068cba70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 -- 172.21.15.106:0/1908677306 >> 172.21.15.106:0/1908677306 conn(0x555b06887a60 msgr2=0x555b068c38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 -- 172.21.15.106:0/1908677306 shutdown_connections 2024-03-28T02:08:56.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 -- 172.21.15.106:0/1908677306 wait complete. 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.225+0000 7f4ef822d740 1 -- start start 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ef822d740 1 Processor -- start 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 0x555b068d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c0130 0x555b0692f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ef822d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b06931960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ef822d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b068d9430 con 0x555b068c0130 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ef822d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b068d8dd0 con 0x555b068c52a0 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ef822d740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b068d9a90 con 0x555b068bf740 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ef822d740 10 client.0 _ll_register_callbacks cb 0x7f4ed8013850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4eeeffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c0130 0x555b0692f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4eeeffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c0130 0x555b0692f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53312/0 (socket says 172.21.15.106:53312) 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4eeeffd640 1 -- 172.21.15.106:0/3124201052 learned_addr learned my addr 172.21.15.106:0/3124201052 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b06931960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 0x555b068d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4eeeffd640 1 -- 172.21.15.106:0/3124201052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 msgr2=0x555b068d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4eeeffd640 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 0x555b068d1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4eeeffd640 1 -- 172.21.15.106:0/3124201052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 msgr2=0x555b06931960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4eeeffd640 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b06931960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4eeeffd640 1 -- 172.21.15.106:0/3124201052 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b06932190 con 0x555b068c0130 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4eeeffd640 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c0130 0x555b0692f2d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4ee002cfd0 tx=0x7f4ee0009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15a9a76a13f1375 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ed67fc640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ed67fc640 1 -- 172.21.15.106:0/3124201052 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00044d0 con 0x555b068c0130 2024-03-28T02:08:56.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ed67fc640 1 -- 172.21.15.106:0/3124201052 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4ee0005040 con 0x555b068c0130 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.226+0000 7f4ef822d740 1 -- 172.21.15.106:0/3124201052 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ed8020410 con 0x555b068c0130 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ed67fc640 1 -- 172.21.15.106:0/3124201052 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ee0053050 con 0x555b068c0130 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ef822d740 10 client.14766 fetch_fsmap learned FSMap version 17 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ef822d740 1 -- 172.21.15.106:0/3124201052 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x555b06933500 con 0x555b068c0130 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ed67fc640 1 -- 172.21.15.106:0/3124201052 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ed8020410 con 0x555b068c0130 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ed67fc640 1 -- 172.21.15.106:0/3124201052 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f4ee00035f0 con 0x555b068c0130 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ef822d740 10 client.14766 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ef822d740 10 client.14766 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ef822d740 20 client.14766 populate_metadata read hostname 'smithi106' 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ef822d740 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x555b06937bf0 0x555b06933840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:56.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ef822d740 4 client.14766 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ef822d740 1 -- 172.21.15.106:0/3124201052 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555b0682c4b0 con 0x555b06937bf0 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x555b06937bf0 0x555b06933840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ee7fff640 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x555b06937bf0 0x555b06933840 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=573f793fb3662c98 server_cookie=9c8acfccc8786460 in_seq=0 out_seq=0 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.227+0000 7f4ed67fc640 10 client.14766 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ed67fc640 1 -- 172.21.15.106:0/3124201052 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x555b0682c4b0 con 0x555b06937bf0 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ed67fc640 10 client.14766 handle_command_reply: tid=0 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 client.14766 shutdown 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 -- 172.21.15.106:0/3124201052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c0130 msgr2=0x555b0692f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c0130 0x555b0692f2d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4ee002cfd0 tx=0x7f4ee0009880 comp rx=0 tx=0).stop 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 -- 172.21.15.106:0/3124201052 shutdown_connections 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x555b06937bf0 0x555b06933840 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b068c52a0 0x555b06931960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b068c0130 0x555b0692f2d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ed67fc640 0 client.14766 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:08:56.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 --2- 172.21.15.106:0/3124201052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555b068bf740 0x555b068d1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 -- 172.21.15.106:0/3124201052 >> 172.21.15.106:0/3124201052 conn(0x555b06887a60 msgr2=0x555b068cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:56.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 -- 172.21.15.106:0/3124201052 shutdown_connections 2024-03-28T02:08:56.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 1 -- 172.21.15.106:0/3124201052 wait complete. 2024-03-28T02:08:56.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef822d740 20 client.14766 trim_cache size 0 max 16384 2024-03-28T02:08:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:08:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 0, 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24509, 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 6.7830937029999996, 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 17, 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 89, 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 13.477986117 2024-03-28T02:08:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 -- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4edc0a2800 msgr2=0x7f4edc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4edc0a2800 0x7f4edc0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8021000 tx=0x7f4ee806e810 comp rx=0 tx=0).stop 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 -- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0102150 msgr2=0x7f4ef019dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0102150 0x7f4ef019dec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800eb10 tx=0x7f4ee800c2d0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 -- 172.21.15.106:0/2013129074 shutdown_connections 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0102150 0x7f4ef019dec0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4edc0a2800 0x7f4edc0a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0101790 0x7f4ef0196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 --2- 172.21.15.106:0/2013129074 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ef01005a0 0x7f4ef0196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:56.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 -- 172.21.15.106:0/2013129074 >> 172.21.15.106:0/2013129074 conn(0x7f4ef00fbb30 msgr2=0x7f4ef0104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:56.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 -- 172.21.15.106:0/2013129074 shutdown_connections 2024-03-28T02:08:56.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:56.228+0000 7f4ef5562640 1 -- 172.21.15.106:0/2013129074 wait complete. 2024-03-28T02:08:56.234 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T02:08:56.234 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T02:08:56.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T02:08:56.471 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:08:56.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:56 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/225023268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:56.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:56 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/499673303' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:56.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:56 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/225023268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:56.737 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:56 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/499673303' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:56.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:56 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/225023268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:08:56.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:56 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/499673303' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:08:57.825 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:57 smithi106 ceph-mon[31965]: pgmap v296: 129 pgs: 4 creating+peering, 120 active+clean, 5 unknown; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s wr, 6 op/s 2024-03-28T02:08:57.825 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:57 smithi106 ceph-mon[31965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T02:08:57.826 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:57 smithi106 ceph-mon[31965]: Cluster is now healthy 2024-03-28T02:08:57.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:57 smithi178 ceph-mon[36521]: pgmap v296: 129 pgs: 4 creating+peering, 120 active+clean, 5 unknown; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s wr, 6 op/s 2024-03-28T02:08:57.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:57 smithi178 ceph-mon[36521]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T02:08:57.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:57 smithi178 ceph-mon[36521]: Cluster is now healthy 2024-03-28T02:08:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:57 smithi153 ceph-mon[36228]: pgmap v296: 129 pgs: 4 creating+peering, 120 active+clean, 5 unknown; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s wr, 6 op/s 2024-03-28T02:08:57.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:57 smithi153 ceph-mon[36228]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T02:08:57.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:57 smithi153 ceph-mon[36228]: Cluster is now healthy 2024-03-28T02:08:58.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.403+0000 7f34551a1700 1 -- 172.21.15.106:0/1740931714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 msgr2=0x7f3450100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.403+0000 7f34551a1700 1 --2- 172.21.15.106:0/1740931714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 0x7f3450100dc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3444009990 tx=0x7f3444009ca0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.404+0000 7f34551a1700 1 -- 172.21.15.106:0/1740931714 shutdown_connections 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.404+0000 7f34551a1700 1 --2- 172.21.15.106:0/1740931714 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3450101300 0x7f345010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.404+0000 7f34551a1700 1 --2- 172.21.15.106:0/1740931714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34501070d0 0x7f34501074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.404+0000 7f34551a1700 1 --2- 172.21.15.106:0/1740931714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 0x7f3450100dc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.404+0000 7f34551a1700 1 -- 172.21.15.106:0/1740931714 >> 172.21.15.106:0/1740931714 conn(0x7f34500762d0 msgr2=0x7f34500766e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.404+0000 7f34551a1700 1 -- 172.21.15.106:0/1740931714 shutdown_connections 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.404+0000 7f34551a1700 1 -- 172.21.15.106:0/1740931714 wait complete. 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.404+0000 7f34551a1700 1 Processor -- start 2024-03-28T02:08:58.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.404+0000 7f34551a1700 1 -- start start 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f34551a1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 0x7f34501a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f34551a1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3450101300 0x7f34501a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f34551a1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34501070d0 0x7f345019a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f34551a1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34501123a0 con 0x7f3450100940 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f34551a1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3450112220 con 0x7f34501070d0 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f34551a1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3450112520 con 0x7f3450101300 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 0x7f34501a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 0x7f34501a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39598/0 (socket says 172.21.15.106:39598) 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344ffff700 1 -- 172.21.15.106:0/2454336525 learned_addr learned my addr 172.21.15.106:0/2454336525 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f34549a0700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34501070d0 0x7f345019a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344ffff700 1 -- 172.21.15.106:0/2454336525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3450101300 msgr2=0x7f34501a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:58.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344f7fe700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3450101300 0x7f34501a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344ffff700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3450101300 0x7f34501a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344ffff700 1 -- 172.21.15.106:0/2454336525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34501070d0 msgr2=0x7f345019a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344ffff700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34501070d0 0x7f345019a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344ffff700 1 -- 172.21.15.106:0/2454336525 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345019afe0 con 0x7f3450100940 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344f7fe700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3450101300 0x7f34501a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344ffff700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 0x7f34501a06c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f344000e950 tx=0x7f344000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344d7fa700 1 -- 172.21.15.106:0/2454336525 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344000eea0 con 0x7f3450100940 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345019b2d0 con 0x7f3450100940 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344d7fa700 1 -- 172.21.15.106:0/2454336525 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3440004510 con 0x7f3450100940 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f344d7fa700 1 -- 172.21.15.106:0/2454336525 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3440010430 con 0x7f3450100940 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.405+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34501a7590 con 0x7f3450100940 2024-03-28T02:08:58.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.406+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34500699d0 con 0x7f3450100940 2024-03-28T02:08:58.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.407+0000 7f344d7fa700 1 -- 172.21.15.106:0/2454336525 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3440003eb0 con 0x7f3450100940 2024-03-28T02:08:58.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.408+0000 7f344d7fa700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34380a2890 0x7f34380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:08:58.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.408+0000 7f344f7fe700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34380a2890 0x7f34380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:08:58.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.408+0000 7f344d7fa700 1 -- 172.21.15.106:0/2454336525 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f34400cef30 con 0x7f3450100940 2024-03-28T02:08:58.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.408+0000 7f344f7fe700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34380a2890 0x7f34380a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f344400b2e0 tx=0x7f3444005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:08:58.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.412+0000 7f344d7fa700 1 -- 172.21.15.106:0/2454336525 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f344009bd90 con 0x7f3450100940 2024-03-28T02:08:58.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.657+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f34501a7760 con 0x7f3450100940 2024-03-28T02:08:58.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.658+0000 7f344d7fa700 1 -- 172.21.15.106:0/2454336525 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f344009b690 con 0x7f3450100940 2024-03-28T02:08:58.651 INFO:teuthology.orchestra.run.smithi106.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T02:08:58.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34380a2890 msgr2=0x7f34380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:58.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34380a2890 0x7f34380a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f344400b2e0 tx=0x7f3444005e80 comp rx=0 tx=0).stop 2024-03-28T02:08:58.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 msgr2=0x7f34501a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:08:58.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 0x7f34501a06c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f344000e950 tx=0x7f344000ec60 comp rx=0 tx=0).stop 2024-03-28T02:08:58.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 shutdown_connections 2024-03-28T02:08:58.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34501070d0 0x7f345019a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34380a2890 0x7f34380a4d50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3450100940 0x7f34501a06c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 --2- 172.21.15.106:0/2454336525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3450101300 0x7f34501a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:08:58.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 >> 172.21.15.106:0/2454336525 conn(0x7f34500762d0 msgr2=0x7f345010d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:08:58.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 shutdown_connections 2024-03-28T02:08:58.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:08:58.660+0000 7f34551a1700 1 -- 172.21.15.106:0/2454336525 wait complete. 2024-03-28T02:08:59.330 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-28T02:08:59.330 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T02:08:59.330 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi106.front.sepia.ceph.com 2024-03-28T02:08:59.330 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-28T02:08:59.330 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T02:08:59.330 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-28T02:08:59.330 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T02:08:59.330 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T02:08:59.330 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T02:08:59.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:08:59.331 DEBUG:teuthology.orchestra.run.smithi106:> ip addr 2024-03-28T02:08:59.347 INFO:teuthology.orchestra.run.smithi106.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T02:08:59.347 INFO:teuthology.orchestra.run.smithi106.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 ::1/128 scope host 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:d8:a1:2a brd ff:ff:ff:ff:ff:ff 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout: altname enp7s0f0 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:d8:a1:2b brd ff:ff:ff:ff:ff:ff 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout: altname enp7s0f1 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:8f:ce:6c brd ff:ff:ff:ff:ff:ff 2024-03-28T02:08:59.348 INFO:teuthology.orchestra.run.smithi106.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T02:08:59.349 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:8f:ce:6d brd ff:ff:ff:ff:ff:ff 2024-03-28T02:08:59.349 INFO:teuthology.orchestra.run.smithi106.stdout: inet 172.21.15.106/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T02:08:59.349 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft 42095sec preferred_lft 42095sec 2024-03-28T02:08:59.349 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 fe80::ec4:7aff:fe8f:ce6d/64 scope link 2024-03-28T02:08:59.349 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-03-28T02:08:59.349 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T02:08:59.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:08:59.350 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:08:59.350 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:08:59.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link add name ceph-brx type bridge 2024-03-28T02:08:59.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip addr flush dev ceph-brx 2024-03-28T02:08:59.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set ceph-brx up 2024-03-28T02:08:59.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T02:08:59.350 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:08:59.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:08:59.519 DEBUG:teuthology.orchestra.run.smithi106:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T02:08:59.546 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-03-28T02:08:59.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:08:59.548 DEBUG:teuthology.orchestra.run.smithi106:> route 2024-03-28T02:08:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:59 smithi106 ceph-mon[31965]: pgmap v297: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2024-03-28T02:08:59.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:08:59 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2454336525' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T02:08:59.632 INFO:teuthology.orchestra.run.smithi106.stdout:Kernel IP routing table 2024-03-28T02:08:59.633 INFO:teuthology.orchestra.run.smithi106.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T02:08:59.633 INFO:teuthology.orchestra.run.smithi106.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T02:08:59.633 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T02:08:59.633 INFO:teuthology.orchestra.run.smithi106.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T02:08:59.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:08:59.633 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:08:59.633 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:08:59.633 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T02:08:59.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T02:08:59.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T02:08:59.634 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:08:59.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:08:59.846 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list 2024-03-28T02:08:59.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:08:59.861 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list-id 2024-03-28T02:08:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:59 smithi178 ceph-mon[36521]: pgmap v297: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2024-03-28T02:08:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:08:59 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2454336525' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T02:08:59.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:08:59.915 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:08:59.915 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:08:59.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T02:08:59.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-28T02:08:59.915 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:08:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:59 smithi153 ceph-mon[36228]: pgmap v297: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2024-03-28T02:08:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:08:59 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2454336525' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T02:08:59.997 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-28T02:08:59.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:08:59.998 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:08:59.998 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:08:59.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-28T02:08:59.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T02:08:59.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-28T02:08:59.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-28T02:08:59.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-28T02:08:59.998 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:09:00.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:09:00.132 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:09:00.132 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:09:00.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set brx.0 up 2024-03-28T02:09:00.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T02:09:00.132 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:09:00.173 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-28T02:09:00.173 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T02:09:00.174 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-28T02:09:00.229 INFO:teuthology.orchestra.run.smithi106.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T02:09:00.229 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T02:09:00.229 DEBUG:teuthology.orchestra.run.smithi106:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-28T02:09:00.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo modprobe fuse 2024-03-28T02:09:00.350 DEBUG:teuthology.orchestra.run.smithi106:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T02:09:00.406 INFO:teuthology.orchestra.run.smithi106.stdout:/proc 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/sys 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/dev 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/kernel/security 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/shm 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/pts 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/run 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/cgroup 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/pstore 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/bpf 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/ 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/selinux 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T02:09:00.407 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/mqueue 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/hugepages 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/kernel/debug 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/kernel/tracing 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/fuse/connections 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/kernel/config 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/run/user/1000 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/ceph 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay 2024-03-28T02:09:00.408 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/7f1e1438838a719d86c727f0128705b7f0719f77645e4eee1a1746548166dda2/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/26cd5f955e157ae69e1adffaa66a44f9f17b444a6d769f2fa49bcda86d484d9b/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/run/user/0 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/228db2431dedf2b95bf430f75377b395a11f38bf4ceddc702c75a793f8cde2a9/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/407af7821ab6b5f4dfa492509ecbb0576e5862f982d43f638c2d34752a3fb182/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/4328b1ce06ea96e042af14a2655648e2ff3eb089c0736ae3817bc0692e1e9e7c/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/f72e54a6282ef0b4ef44acee0e2ae797526190edd0c1fb480de8ece4a28a0334/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/dad1a5ba248356baccd7c2f200089fe05444c925359568acdfd9510a6fcbcf3e/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/affecec514067e05f5d2805a65059a20f1ef140f2cd3e4f10518300e8cde100a/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/4a3ae991211b047e2fbca06ceef43acff470245d5870aafa96c7bab1f8faa287/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/b0a0909183121915b73235546f0c43250ec17fc7b1c8e822656c611d9daf0683/merged 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/run/netns 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T02:09:00.409 INFO:teuthology.orchestra.run.smithi106.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T02:09:00.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:09:00.410 DEBUG:teuthology.orchestra.run.smithi106:> ls /sys/fs/fuse/connections 2024-03-28T02:09:00.461 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T02:09:00.462 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-28T02:09:00.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo modprobe fuse 2024-03-28T02:09:00.530 DEBUG:teuthology.orchestra.run.smithi106:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/proc 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/sys 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/dev 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/kernel/security 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/shm 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/pts 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/run 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/cgroup 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/pstore 2024-03-28T02:09:00.587 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/bpf 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/ 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/selinux 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/mqueue 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/hugepages 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/kernel/debug 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/kernel/tracing 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/fs/fuse/connections 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/sys/kernel/config 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/run/user/1000 2024-03-28T02:09:00.588 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/ceph 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/7f1e1438838a719d86c727f0128705b7f0719f77645e4eee1a1746548166dda2/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/26cd5f955e157ae69e1adffaa66a44f9f17b444a6d769f2fa49bcda86d484d9b/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/run/user/0 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/228db2431dedf2b95bf430f75377b395a11f38bf4ceddc702c75a793f8cde2a9/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/407af7821ab6b5f4dfa492509ecbb0576e5862f982d43f638c2d34752a3fb182/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/4328b1ce06ea96e042af14a2655648e2ff3eb089c0736ae3817bc0692e1e9e7c/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/f72e54a6282ef0b4ef44acee0e2ae797526190edd0c1fb480de8ece4a28a0334/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/dad1a5ba248356baccd7c2f200089fe05444c925359568acdfd9510a6fcbcf3e/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/affecec514067e05f5d2805a65059a20f1ef140f2cd3e4f10518300e8cde100a/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/4a3ae991211b047e2fbca06ceef43acff470245d5870aafa96c7bab1f8faa287/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/var/lib/containers/storage/overlay/b0a0909183121915b73235546f0c43250ec17fc7b1c8e822656c611d9daf0683/merged 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/run/netns 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T02:09:00.589 INFO:teuthology.orchestra.run.smithi106.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T02:09:00.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:09:00.590 DEBUG:teuthology.orchestra.run.smithi106:> ls /sys/fs/fuse/connections 2024-03-28T02:09:00.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi106.stderr:2024-03-28T02:09:00.607+0000 7fe085e35480 -1 init, newargv = 0x55aee08e8ca0 newargc=15 2024-03-28T02:09:00.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi106.stderr:ceph-fuse[76246]: starting ceph client 2024-03-28T02:09:00.606 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi106.stderr:ceph-fuse[76246]: starting fuse 2024-03-28T02:09:00.615 INFO:teuthology.orchestra.run.smithi106.stdout:119 2024-03-28T02:09:00.615 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo stdin-killer -- python3 -c ' 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> import glob 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> import re 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> import os 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> import subprocess 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> def _find_admin_socket(client_name): 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> files = glob.glob(asok_path) 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> # Given a non-glob path, it better be there 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> if "*" not in asok_path: 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> assert(len(files) == 1) 2024-03-28T02:09:00.616 DEBUG:teuthology.orchestra.run.smithi106:> return files[0] 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> for f in files: 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> contents = proc_f.read() 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> if mountpoint in contents: 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> return f 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> print(_find_admin_socket("client.admin")) 2024-03-28T02:09:00.617 DEBUG:teuthology.orchestra.run.smithi106:> ' 2024-03-28T02:09:00.736 INFO:teuthology.orchestra.run.smithi106.stdout:/var/run/ceph/ceph-client.admin.76246.asok 2024-03-28T02:09:00.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T02:09:00.747 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76246.asok 2024-03-28T02:09:00.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:09:00.747 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76246.asok status 2024-03-28T02:09:00.884 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "metadata": { 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "entity_id": "admin", 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "hostname": "smithi106", 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "pid": "76246", 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "root": "/" 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "dentry_count": 0, 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "dentry_pinned_count": 0, 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 14778, 2024-03-28T02:09:00.885 INFO:teuthology.orchestra.run.smithi106.stdout: "inst": { 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "name": { 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "client", 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "num": 14778 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": { 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "192.168.0.1:0", 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1376955996 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": { 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "192.168.0.1:0", 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1376955996 2024-03-28T02:09:00.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:09:00.887 INFO:teuthology.orchestra.run.smithi106.stdout: "inst_str": "client.14778 192.168.0.1:0/1376955996", 2024-03-28T02:09:00.887 INFO:teuthology.orchestra.run.smithi106.stdout: "addr_str": "192.168.0.1:0/1376955996", 2024-03-28T02:09:00.887 INFO:teuthology.orchestra.run.smithi106.stdout: "inode_count": 1, 2024-03-28T02:09:00.887 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_epoch": 17, 2024-03-28T02:09:00.887 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_epoch": 89, 2024-03-28T02:09:00.887 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_epoch_barrier": 0, 2024-03-28T02:09:00.887 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklisted": false, 2024-03-28T02:09:00.887 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs" 2024-03-28T02:09:00.887 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:09:00.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:09:00.893 DEBUG:teuthology.orchestra.run.smithi106:> stat --file-system '--printf=%T 2024-03-28T02:09:00.893 DEBUG:teuthology.orchestra.run.smithi106:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-28T02:09:00.908 INFO:teuthology.orchestra.run.smithi106.stdout:fuseblk 2024-03-28T02:09:00.908 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-28T02:09:00.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:09:00.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-28T02:09:00.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:09:00.976 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-28T02:09:01.051 INFO:teuthology.orchestra.run.smithi106.stdout:# file: . 2024-03-28T02:09:01.051 INFO:teuthology.orchestra.run.smithi106.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-28T02:09:01.052 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:01.052 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:09:01.052 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T02:09:01.108 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 -- 172.21.15.106:0/1861684479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 msgr2=0x7fee8c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 --2- 172.21.15.106:0/1861684479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 0x7fee8c106e60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fee7c009900 tx=0x7fee7c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 -- 172.21.15.106:0/1861684479 shutdown_connections 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 --2- 172.21.15.106:0/1861684479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee8c1073a0 0x7fee8c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 --2- 172.21.15.106:0/1861684479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 0x7fee8c106e60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 --2- 172.21.15.106:0/1861684479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee8c102140 0x7fee8c104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 -- 172.21.15.106:0/1861684479 >> 172.21.15.106:0/1861684479 conn(0x7fee8c0fbb30 msgr2=0x7fee8c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 -- 172.21.15.106:0/1861684479 shutdown_connections 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 -- 172.21.15.106:0/1861684479 wait complete. 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.256+0000 7fee947f4640 1 Processor -- start 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee947f4640 1 -- start start 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee947f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee8c102140 0x7fee8c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee947f4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 0x7fee8c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee947f4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee8c1073a0 0x7fee8c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee947f4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c10d330 con 0x7fee8c104a70 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee947f4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c10d1b0 con 0x7fee8c102140 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee947f4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c10d4b0 con 0x7fee8c1073a0 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee91d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 0x7fee8c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee91d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 0x7fee8c197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39620/0 (socket says 172.21.15.106:39620) 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee91d8f640 1 -- 172.21.15.106:0/3183153150 learned_addr learned my addr 172.21.15.106:0/3183153150 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee92590640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee8c102140 0x7fee8c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee92d91640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee8c1073a0 0x7fee8c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee91d8f640 1 -- 172.21.15.106:0/3183153150 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee8c1073a0 msgr2=0x7fee8c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee91d8f640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee8c1073a0 0x7fee8c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee91d8f640 1 -- 172.21.15.106:0/3183153150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee8c102140 msgr2=0x7fee8c197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee91d8f640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee8c102140 0x7fee8c197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee91d8f640 1 -- 172.21.15.106:0/3183153150 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c19f630 con 0x7fee8c104a70 2024-03-28T02:09:01.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee91d8f640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 0x7fee8c197f50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fee7c000c00 tx=0x7fee7c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:01.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee837fe640 1 -- 172.21.15.106:0/3183153150 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c041070 con 0x7fee8c104a70 2024-03-28T02:09:01.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee837fe640 1 -- 172.21.15.106:0/3183153150 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee7c0044d0 con 0x7fee8c104a70 2024-03-28T02:09:01.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c19f8c0 con 0x7fee8c104a70 2024-03-28T02:09:01.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.257+0000 7fee837fe640 1 -- 172.21.15.106:0/3183153150 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c034440 con 0x7fee8c104a70 2024-03-28T02:09:01.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.258+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c19fdb0 con 0x7fee8c104a70 2024-03-28T02:09:01.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.258+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c0fd620 con 0x7fee8c104a70 2024-03-28T02:09:01.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.260+0000 7fee837fe640 1 -- 172.21.15.106:0/3183153150 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fee7c044070 con 0x7fee8c104a70 2024-03-28T02:09:01.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.260+0000 7fee837fe640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fee780a2800 0x7fee780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.260+0000 7fee92590640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fee780a2800 0x7fee780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.260+0000 7fee837fe640 1 -- 172.21.15.106:0/3183153150 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fee7c0f0a60 con 0x7fee8c104a70 2024-03-28T02:09:01.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.261+0000 7fee92590640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fee780a2800 0x7fee780a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee84005fd0 tx=0x7fee84009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:01.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.264+0000 7fee837fe640 1 -- 172.21.15.106:0/3183153150 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee7c0bd8c0 con 0x7fee8c104a70 2024-03-28T02:09:01.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.441+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee8c02d1f0 con 0x7fee8c104a70 2024-03-28T02:09:01.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.443+0000 7fee837fe640 1 -- 172.21.15.106:0/3183153150 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7fee7c0bd1c0 con 0x7fee8c104a70 2024-03-28T02:09:01.437 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:01.437 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fee780a2800 msgr2=0x7fee780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fee780a2800 0x7fee780a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee84005fd0 tx=0x7fee84009500 comp rx=0 tx=0).stop 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 msgr2=0x7fee8c197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 0x7fee8c197f50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fee7c000c00 tx=0x7fee7c005d50 comp rx=0 tx=0).stop 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 shutdown_connections 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee8c1073a0 0x7fee8c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee8c104a70 0x7fee8c197f50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fee780a2800 0x7fee780a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 --2- 172.21.15.106:0/3183153150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee8c102140 0x7fee8c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 >> 172.21.15.106:0/3183153150 conn(0x7fee8c0fbb30 msgr2=0x7fee8c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.445+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 shutdown_connections 2024-03-28T02:09:01.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.446+0000 7fee947f4640 1 -- 172.21.15.106:0/3183153150 wait complete. 2024-03-28T02:09:01.451 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T02:08:46.952662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T02:08:47.976273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T02:08:51.703592+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T02:09:01.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.565+0000 7fe447e9a640 1 -- 172.21.15.106:0/4128624718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440100880 msgr2=0x7fe440100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.565+0000 7fe447e9a640 1 --2- 172.21.15.106:0/4128624718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440100880 0x7fe440100d00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe430009900 tx=0x7fe43002bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.565+0000 7fe447e9a640 1 -- 172.21.15.106:0/4128624718 shutdown_connections 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.565+0000 7fe447e9a640 1 --2- 172.21.15.106:0/4128624718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe440101240 0x7fe440108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.565+0000 7fe447e9a640 1 --2- 172.21.15.106:0/4128624718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440100880 0x7fe440100d00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.565+0000 7fe447e9a640 1 --2- 172.21.15.106:0/4128624718 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe440101b50 0x7fe440101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.565+0000 7fe447e9a640 1 -- 172.21.15.106:0/4128624718 >> 172.21.15.106:0/4128624718 conn(0x7fe4400fb2b0 msgr2=0x7fe4400fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.565+0000 7fe447e9a640 1 -- 172.21.15.106:0/4128624718 shutdown_connections 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.565+0000 7fe447e9a640 1 -- 172.21.15.106:0/4128624718 wait complete. 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe447e9a640 1 Processor -- start 2024-03-28T02:09:01.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe447e9a640 1 -- start start 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe447e9a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe440100880 0x7fe4401002e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe447e9a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440101240 0x7fe44019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe447e9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe440101b50 0x7fe44019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe447e9a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44010c260 con 0x7fe440101240 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe447e9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44010c0e0 con 0x7fe440101b50 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe447e9a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44010c3e0 con 0x7fe440100880 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe445435640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440101240 0x7fe44019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe445435640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440101240 0x7fe44019c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39638/0 (socket says 172.21.15.106:39638) 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe445435640 1 -- 172.21.15.106:0/3135361755 learned_addr learned my addr 172.21.15.106:0/3135361755 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe446437640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe440101b50 0x7fe44019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe445435640 1 -- 172.21.15.106:0/3135361755 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe440100880 msgr2=0x7fe4401002e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe445435640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe440100880 0x7fe4401002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe445435640 1 -- 172.21.15.106:0/3135361755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe440101b50 msgr2=0x7fe44019ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe445435640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe440101b50 0x7fe44019ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.566+0000 7fe445435640 1 -- 172.21.15.106:0/3135361755 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44019f5c0 con 0x7fe440101240 2024-03-28T02:09:01.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.567+0000 7fe445435640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440101240 0x7fe44019c690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe43002efd0 tx=0x7fe430005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:01.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.567+0000 7fe436ffd640 1 -- 172.21.15.106:0/3135361755 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe430041070 con 0x7fe440101240 2024-03-28T02:09:01.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.567+0000 7fe436ffd640 1 -- 172.21.15.106:0/3135361755 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4300044d0 con 0x7fe440101240 2024-03-28T02:09:01.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.567+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44019f7f0 con 0x7fe440101240 2024-03-28T02:09:01.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.567+0000 7fe436ffd640 1 -- 172.21.15.106:0/3135361755 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe430034440 con 0x7fe440101240 2024-03-28T02:09:01.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.567+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44019fce0 con 0x7fe440101240 2024-03-28T02:09:01.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.568+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4401079c0 con 0x7fe440101240 2024-03-28T02:09:01.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.569+0000 7fe436ffd640 1 -- 172.21.15.106:0/3135361755 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe430044070 con 0x7fe440101240 2024-03-28T02:09:01.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.569+0000 7fe436ffd640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe42c0a2800 0x7fe42c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.570+0000 7fe445c36640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe42c0a2800 0x7fe42c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.570+0000 7fe436ffd640 1 -- 172.21.15.106:0/3135361755 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe4300f0a40 con 0x7fe440101240 2024-03-28T02:09:01.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.570+0000 7fe445c36640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe42c0a2800 0x7fe42c0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe440069a10 tx=0x7fe43c005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:01.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.573+0000 7fe436ffd640 1 -- 172.21.15.106:0/3135361755 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4300bd8a0 con 0x7fe440101240 2024-03-28T02:09:01.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.748+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe44002d200 con 0x7fe440101240 2024-03-28T02:09:01.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.748+0000 7fe436ffd640 1 -- 172.21.15.106:0/3135361755 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe4300bd1a0 con 0x7fe440101240 2024-03-28T02:09:01.741 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe42c0a2800 msgr2=0x7fe42c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe42c0a2800 0x7fe42c0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe440069a10 tx=0x7fe43c005c90 comp rx=0 tx=0).stop 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440101240 msgr2=0x7fe44019c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440101240 0x7fe44019c690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe43002efd0 tx=0x7fe430005d50 comp rx=0 tx=0).stop 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 shutdown_connections 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe42c0a2800 0x7fe42c0a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe440101b50 0x7fe44019ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe440101240 0x7fe44019c690 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 --2- 172.21.15.106:0/3135361755 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe440100880 0x7fe4401002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 >> 172.21.15.106:0/3135361755 conn(0x7fe4400fb2b0 msgr2=0x7fe440104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 shutdown_connections 2024-03-28T02:09:01.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.750+0000 7fe447e9a640 1 -- 172.21.15.106:0/3135361755 wait complete. 2024-03-28T02:09:01.753 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:01 smithi106 ceph-mon[31965]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 519 B/s wr, 1 op/s 2024-03-28T02:09:01.753 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:01 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3183153150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:01.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T02:09:01.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.865+0000 7feef77f4640 1 -- 172.21.15.106:0/1127583473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef0102150 msgr2=0x7feef01089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.865+0000 7feef77f4640 1 --2- 172.21.15.106:0/1127583473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef0102150 0x7feef01089e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feee8009ca0 tx=0x7feee802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.865+0000 7feef77f4640 1 -- 172.21.15.106:0/1127583473 shutdown_connections 2024-03-28T02:09:01.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.865+0000 7feef77f4640 1 --2- 172.21.15.106:0/1127583473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef0102150 0x7feef01089e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.865+0000 7feef77f4640 1 --2- 172.21.15.106:0/1127583473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef0101790 0x7feef0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.865+0000 7feef77f4640 1 --2- 172.21.15.106:0/1127583473 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feef01005a0 0x7feef01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 -- 172.21.15.106:0/1127583473 >> 172.21.15.106:0/1127583473 conn(0x7feef00fbb30 msgr2=0x7feef00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 -- 172.21.15.106:0/1127583473 shutdown_connections 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 -- 172.21.15.106:0/1127583473 wait complete. 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 Processor -- start 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 -- start start 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef01005a0 0x7feef01979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feef0101790 0x7feef0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef0102150 0x7feef019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef010c230 con 0x7feef01005a0 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef010c0b0 con 0x7feef0102150 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef77f4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef010c3b0 con 0x7feef0101790 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef5590640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef01005a0 0x7feef01979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef5590640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef01005a0 0x7feef01979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39660/0 (socket says 172.21.15.106:39660) 2024-03-28T02:09:01.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef5590640 1 -- 172.21.15.106:0/4046562753 learned_addr learned my addr 172.21.15.106:0/4046562753 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.866+0000 7feef5d91640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef0102150 0x7feef019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feef4d8f640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feef0101790 0x7feef0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feef5590640 1 -- 172.21.15.106:0/4046562753 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feef0101790 msgr2=0x7feef0197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feef5590640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feef0101790 0x7feef0197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feef5590640 1 -- 172.21.15.106:0/4046562753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef0102150 msgr2=0x7feef019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feef5590640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef0102150 0x7feef019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feef5590640 1 -- 172.21.15.106:0/4046562753 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef019f5f0 con 0x7feef01005a0 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feef5590640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef01005a0 0x7feef01979d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feeec00e9c0 tx=0x7feeec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feee67fc640 1 -- 172.21.15.106:0/4046562753 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec00cd80 con 0x7feef01005a0 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feee67fc640 1 -- 172.21.15.106:0/4046562753 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feeec0035f0 con 0x7feef01005a0 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef019f8e0 con 0x7feef01005a0 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feee67fc640 1 -- 172.21.15.106:0/4046562753 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec010690 con 0x7feef01005a0 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.867+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef019fe30 con 0x7feef01005a0 2024-03-28T02:09:01.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.868+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef01a00c0 con 0x7feef01005a0 2024-03-28T02:09:01.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.869+0000 7feee67fc640 1 -- 172.21.15.106:0/4046562753 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7feeec0037a0 con 0x7feef01005a0 2024-03-28T02:09:01.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.870+0000 7feee67fc640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feedc0a2880 0x7feedc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:01.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.870+0000 7feef4d8f640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feedc0a2880 0x7feedc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:01.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.870+0000 7feee67fc640 1 -- 172.21.15.106:0/4046562753 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7feeec0cf020 con 0x7feef01005a0 2024-03-28T02:09:01.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.870+0000 7feef4d8f640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feedc0a2880 0x7feedc0a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feef0198ee0 tx=0x7feee0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:01.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:01.874+0000 7feee67fc640 1 -- 172.21.15.106:0/4046562753 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feeec09be80 con 0x7feef01005a0 2024-03-28T02:09:01.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:01 smithi178 ceph-mon[36521]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 519 B/s wr, 1 op/s 2024-03-28T02:09:01.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:01 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3183153150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:01 smithi153 ceph-mon[36228]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 519 B/s wr, 1 op/s 2024-03-28T02:09:01.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:01 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3183153150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:02.040 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.047+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feef01a0290 con 0x7feef01005a0 2024-03-28T02:09:02.041 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.047+0000 7feee67fc640 1 -- 172.21.15.106:0/4046562753 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7feeec09b780 con 0x7feef01005a0 2024-03-28T02:09:02.041 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feedc0a2880 msgr2=0x7feedc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feedc0a2880 0x7feedc0a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feef0198ee0 tx=0x7feee0005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef01005a0 msgr2=0x7feef01979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef01005a0 0x7feef01979d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feeec00e9c0 tx=0x7feeec00ee80 comp rx=0 tx=0).stop 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 shutdown_connections 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feedc0a2880 0x7feedc0a4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef0102150 0x7feef019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feef0101790 0x7feef0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 --2- 172.21.15.106:0/4046562753 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feef01005a0 0x7feef01979d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 >> 172.21.15.106:0/4046562753 conn(0x7feef00fbb30 msgr2=0x7feef00fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:02.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.049+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 shutdown_connections 2024-03-28T02:09:02.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.050+0000 7feef77f4640 1 -- 172.21.15.106:0/4046562753 wait complete. 2024-03-28T02:09:02.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 -- 172.21.15.106:0/3206311609 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df01021b0 msgr2=0x7f4df01045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/3206311609 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df01021b0 0x7f4df01045a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009900 tx=0x7f4de002bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 -- 172.21.15.106:0/3206311609 shutdown_connections 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/3206311609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0107410 0x7f4df0109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/3206311609 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df0104ae0 0x7f4df0106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/3206311609 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df01021b0 0x7f4df01045a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 -- 172.21.15.106:0/3206311609 >> 172.21.15.106:0/3206311609 conn(0x7f4df00fbb40 msgr2=0x7f4df00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 -- 172.21.15.106:0/3206311609 shutdown_connections 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 -- 172.21.15.106:0/3206311609 wait complete. 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.164+0000 7f4df4e7e640 1 Processor -- start 2024-03-28T02:09:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4df4e7e640 1 -- start start 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4df4e7e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df01021b0 0x7f4df0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4df4e7e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df0104ae0 0x7f4df0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4df4e7e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0107410 0x7f4df019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4df4e7e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df010d330 con 0x7f4df0104ae0 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4df4e7e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df010d1b0 con 0x7f4df0107410 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4df4e7e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df010d4b0 con 0x7f4df01021b0 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dedd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df0104ae0 0x7f4df0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dedd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df0104ae0 0x7f4df0197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39674/0 (socket says 172.21.15.106:39674) 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dedd9b640 1 -- 172.21.15.106:0/186580138 learned_addr learned my addr 172.21.15.106:0/186580138 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4deed9d640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0107410 0x7f4df019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dee59c640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df01021b0 0x7f4df0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dedd9b640 1 -- 172.21.15.106:0/186580138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df01021b0 msgr2=0x7f4df0197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dedd9b640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df01021b0 0x7f4df0197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dedd9b640 1 -- 172.21.15.106:0/186580138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0107410 msgr2=0x7f4df019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dedd9b640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0107410 0x7f4df019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dedd9b640 1 -- 172.21.15.106:0/186580138 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df019f650 con 0x7f4df0104ae0 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dee59c640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df01021b0 0x7f4df0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4dedd9b640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df0104ae0 0x7f4df0197f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4de400c970 tx=0x7f4de400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4ddf7fe640 1 -- 172.21.15.106:0/186580138 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de4007bb0 con 0x7f4df0104ae0 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4ddf7fe640 1 -- 172.21.15.106:0/186580138 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4de4007d50 con 0x7f4df0104ae0 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.165+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df019f940 con 0x7f4df0104ae0 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.166+0000 7f4ddf7fe640 1 -- 172.21.15.106:0/186580138 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de400f450 con 0x7f4df0104ae0 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.166+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df019fe90 con 0x7f4df0104ae0 2024-03-28T02:09:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.167+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df00fd630 con 0x7f4df0104ae0 2024-03-28T02:09:02.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.167+0000 7f4ddf7fe640 1 -- 172.21.15.106:0/186580138 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4de4004ee0 con 0x7f4df0104ae0 2024-03-28T02:09:02.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.168+0000 7f4ddf7fe640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.168+0000 7f4dee59c640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.168+0000 7f4ddf7fe640 1 -- 172.21.15.106:0/186580138 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4de40ced10 con 0x7f4df0104ae0 2024-03-28T02:09:02.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.169+0000 7f4dee59c640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4de00096c0 tx=0x7f4de0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:02.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.173+0000 7f4ddf7fe640 1 -- 172.21.15.106:0/186580138 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4de4014030 con 0x7f4df0104ae0 2024-03-28T02:09:02.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.349+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4df00622a0 con 0x7f4df0104ae0 2024-03-28T02:09:02.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.349+0000 7f4ddf7fe640 1 -- 172.21.15.106:0/186580138 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4de409bb70 con 0x7f4df0104ae0 2024-03-28T02:09:02.343 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 64 2024-03-28T02:09:02.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4dd80a2800 msgr2=0x7f4dd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4de00096c0 tx=0x7f4de0000bc0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df0104ae0 msgr2=0x7f4df0197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df0104ae0 0x7f4df0197f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4de400c970 tx=0x7f4de400cd70 comp rx=0 tx=0).stop 2024-03-28T02:09:02.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 shutdown_connections 2024-03-28T02:09:02.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0107410 0x7f4df019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df0104ae0 0x7f4df0197f70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 --2- 172.21.15.106:0/186580138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df01021b0 0x7f4df0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 >> 172.21.15.106:0/186580138 conn(0x7f4df00fbb40 msgr2=0x7f4df0102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:02.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 shutdown_connections 2024-03-28T02:09:02.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.351+0000 7f4df4e7e640 1 -- 172.21.15.106:0/186580138 wait complete. 2024-03-28T02:09:02.355 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T02:09:02.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.466+0000 7fa17a097640 1 -- 172.21.15.106:0/4025509192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 msgr2=0x7fa1741009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.466+0000 7fa17a097640 1 --2- 172.21.15.106:0/4025509192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 0x7fa1741009c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa164009900 tx=0x7fa16402bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 -- 172.21.15.106:0/4025509192 shutdown_connections 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 --2- 172.21.15.106:0/4025509192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa174102150 0x7fa1741089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 --2- 172.21.15.106:0/4025509192 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa174101790 0x7fa174101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 --2- 172.21.15.106:0/4025509192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 0x7fa1741009c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 -- 172.21.15.106:0/4025509192 >> 172.21.15.106:0/4025509192 conn(0x7fa1740fbb30 msgr2=0x7fa1740fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 -- 172.21.15.106:0/4025509192 shutdown_connections 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 -- 172.21.15.106:0/4025509192 wait complete. 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 Processor -- start 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 -- start start 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 0x7fa174197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa174101790 0x7fa174197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa174102150 0x7fa17419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17410d300 con 0x7fa1741005a0 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17410d180 con 0x7fa174102150 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.467+0000 7fa17a097640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17410d480 con 0x7fa174101790 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa1737fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 0x7fa174197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa1737fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 0x7fa174197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39682/0 (socket says 172.21.15.106:39682) 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa1737fe640 1 -- 172.21.15.106:0/1998325276 learned_addr learned my addr 172.21.15.106:0/1998325276 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa173fff640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa174102150 0x7fa17419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa172ffd640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa174101790 0x7fa174197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa1737fe640 1 -- 172.21.15.106:0/1998325276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa174101790 msgr2=0x7fa174197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa1737fe640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa174101790 0x7fa174197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa1737fe640 1 -- 172.21.15.106:0/1998325276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa174102150 msgr2=0x7fa17419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa1737fe640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa174102150 0x7fa17419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa1737fe640 1 -- 172.21.15.106:0/1998325276 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17419f630 con 0x7fa1741005a0 2024-03-28T02:09:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa173fff640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa174102150 0x7fa17419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa1737fe640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 0x7fa174197a10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa164000c00 tx=0x7fa164005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa170ff9640 1 -- 172.21.15.106:0/1998325276 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa164041070 con 0x7fa1741005a0 2024-03-28T02:09:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa170ff9640 1 -- 172.21.15.106:0/1998325276 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1640044d0 con 0x7fa1741005a0 2024-03-28T02:09:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17419f8c0 con 0x7fa1741005a0 2024-03-28T02:09:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa170ff9640 1 -- 172.21.15.106:0/1998325276 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa164034440 con 0x7fa1741005a0 2024-03-28T02:09:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.468+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17419fdb0 con 0x7fa1741005a0 2024-03-28T02:09:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.469+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1741079b0 con 0x7fa1741005a0 2024-03-28T02:09:02.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.471+0000 7fa170ff9640 1 -- 172.21.15.106:0/1998325276 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa164044070 con 0x7fa1741005a0 2024-03-28T02:09:02.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.472+0000 7fa170ff9640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa15c0a2800 0x7fa15c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.472+0000 7fa172ffd640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa15c0a2800 0x7fa15c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.472+0000 7fa170ff9640 1 -- 172.21.15.106:0/1998325276 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa1640f09b0 con 0x7fa1741005a0 2024-03-28T02:09:02.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.472+0000 7fa172ffd640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa15c0a2800 0x7fa15c0a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa174198f20 tx=0x7fa16800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.475+0000 7fa170ff9640 1 -- 172.21.15.106:0/1998325276 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa1640bd1c0 con 0x7fa1741005a0 2024-03-28T02:09:02.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.653+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa17402d2c0 con 0x7fa1741005a0 2024-03-28T02:09:02.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.654+0000 7fa170ff9640 1 -- 172.21.15.106:0/1998325276 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa1640c1260 con 0x7fa1741005a0 2024-03-28T02:09:02.647 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:09:02.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.655+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa15c0a2800 msgr2=0x7fa15c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.655+0000 7fa17a097640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa15c0a2800 0x7fa15c0a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa174198f20 tx=0x7fa16800b390 comp rx=0 tx=0).stop 2024-03-28T02:09:02.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 msgr2=0x7fa174197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 0x7fa174197a10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa164000c00 tx=0x7fa164005d50 comp rx=0 tx=0).stop 2024-03-28T02:09:02.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 shutdown_connections 2024-03-28T02:09:02.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa15c0a2800 0x7fa15c0a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa174102150 0x7fa17419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa174101790 0x7fa174197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 --2- 172.21.15.106:0/1998325276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741005a0 0x7fa174197a10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 >> 172.21.15.106:0/1998325276 conn(0x7fa1740fbb30 msgr2=0x7fa174104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:02.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 shutdown_connections 2024-03-28T02:09:02.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.656+0000 7fa17a097640 1 -- 172.21.15.106:0/1998325276 wait complete. 2024-03-28T02:09:02.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3135361755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:09:02.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4046562753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:09:02.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/186580138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:09:02.660 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T02:09:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.769+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/4178688879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 msgr2=0x7fe29c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.769+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/4178688879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 0x7fe29c101c30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009900 tx=0x7fe28c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.769+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/4178688879 shutdown_connections 2024-03-28T02:09:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.769+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/4178688879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe29c102170 0x7fe29c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.769+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/4178688879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 0x7fe29c101c30 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.769+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/4178688879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe29c1005c0 0x7fe29c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.769+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/4178688879 >> 172.21.15.106:0/4178688879 conn(0x7fe29c0fbb30 msgr2=0x7fe29c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.769+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/4178688879 shutdown_connections 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.769+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/4178688879 wait complete. 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a3b9d640 1 Processor -- start 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a3b9d640 1 -- start start 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a3b9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe29c1005c0 0x7fe29c06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a3b9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 0x7fe29c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a3b9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe29c102170 0x7fe29c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a3b9d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c10c2b0 con 0x7fe29c1017b0 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a3b9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c10c130 con 0x7fe29c102170 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a3b9d640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c10c430 con 0x7fe29c1005c0 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a1138640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 0x7fe29c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a1138640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 0x7fe29c19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39700/0 (socket says 172.21.15.106:39700) 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a1138640 1 -- 172.21.15.106:0/1031565169 learned_addr learned my addr 172.21.15.106:0/1031565169 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a213a640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe29c102170 0x7fe29c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a1939640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe29c1005c0 0x7fe29c06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a1138640 1 -- 172.21.15.106:0/1031565169 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe29c1005c0 msgr2=0x7fe29c06a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a1138640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe29c1005c0 0x7fe29c06a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a1138640 1 -- 172.21.15.106:0/1031565169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe29c102170 msgr2=0x7fe29c19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a1138640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe29c102170 0x7fe29c19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.770+0000 7fe2a1138640 1 -- 172.21.15.106:0/1031565169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c19f730 con 0x7fe29c1017b0 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.771+0000 7fe2a1138640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 0x7fe29c19c800 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe28c006010 tx=0x7fe28c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.771+0000 7fe292ffd640 1 -- 172.21.15.106:0/1031565169 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c041070 con 0x7fe29c1017b0 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.771+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c19f960 con 0x7fe29c1017b0 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.771+0000 7fe292ffd640 1 -- 172.21.15.106:0/1031565169 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe28c0044d0 con 0x7fe29c1017b0 2024-03-28T02:09:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.771+0000 7fe292ffd640 1 -- 172.21.15.106:0/1031565169 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c034440 con 0x7fe29c1017b0 2024-03-28T02:09:02.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.771+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c19fe50 con 0x7fe29c1017b0 2024-03-28T02:09:02.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.772+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c107980 con 0x7fe29c1017b0 2024-03-28T02:09:02.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.773+0000 7fe292ffd640 1 -- 172.21.15.106:0/1031565169 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe28c044070 con 0x7fe29c1017b0 2024-03-28T02:09:02.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.773+0000 7fe292ffd640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2880a2800 0x7fe2880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:02.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.774+0000 7fe2a1939640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2880a2800 0x7fe2880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:02.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.774+0000 7fe292ffd640 1 -- 172.21.15.106:0/1031565169 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe28c0f0910 con 0x7fe29c1017b0 2024-03-28T02:09:02.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.774+0000 7fe2a1939640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2880a2800 0x7fe2880a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe29c101610 tx=0x7fe298008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:02.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.778+0000 7fe292ffd640 1 -- 172.21.15.106:0/1031565169 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe28c0bd770 con 0x7fe29c1017b0 2024-03-28T02:09:02.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3135361755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:09:02.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4046562753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:09:02.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/186580138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:09:02.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.966+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe29c0622a0 con 0x7fe29c1017b0 2024-03-28T02:09:02.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.966+0000 7fe292ffd640 1 -- 172.21.15.106:0/1031565169 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+596 (secure 0 0 0) 0x7fe28c0bd070 con 0x7fe29c1017b0 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/11266040 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/2884389364 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/4050392782 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6800/1510653090 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/103827077 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6801/305853075 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/256234961 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6800/305853075 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6801/1510653090 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/2759478196 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:02.960 INFO:teuthology.orchestra.run.smithi106.stderr:listed 10 entries 2024-03-28T02:09:02.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2880a2800 msgr2=0x7fe2880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2880a2800 0x7fe2880a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe29c101610 tx=0x7fe298008040 comp rx=0 tx=0).stop 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 msgr2=0x7fe29c19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 0x7fe29c19c800 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe28c006010 tx=0x7fe28c004ab0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 shutdown_connections 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2880a2800 0x7fe2880a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe29c102170 0x7fe29c19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe29c1017b0 0x7fe29c19c800 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 --2- 172.21.15.106:0/1031565169 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe29c1005c0 0x7fe29c06a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 >> 172.21.15.106:0/1031565169 conn(0x7fe29c0fbb30 msgr2=0x7fe29c104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 shutdown_connections 2024-03-28T02:09:02.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:02.968+0000 7fe2a3b9d640 1 -- 172.21.15.106:0/1031565169 wait complete. 2024-03-28T02:09:02.972 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:09:02.972 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T02:09:02.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3135361755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:09:02.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4046562753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:09:02.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/186580138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:09:02.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:09:03.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 -- 172.21.15.106:0/878863623 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 msgr2=0x7f67b4106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 --2- 172.21.15.106:0/878863623 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 0x7f67b4106ec0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009900 tx=0x7f67a402bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:03.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 -- 172.21.15.106:0/878863623 shutdown_connections 2024-03-28T02:09:03.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 --2- 172.21.15.106:0/878863623 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b4107400 0x7f67b41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 --2- 172.21.15.106:0/878863623 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 0x7f67b4106ec0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 --2- 172.21.15.106:0/878863623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b41021a0 0x7f67b4104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 -- 172.21.15.106:0/878863623 >> 172.21.15.106:0/878863623 conn(0x7f67b40fbb30 msgr2=0x7f67b40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 -- 172.21.15.106:0/878863623 shutdown_connections 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 -- 172.21.15.106:0/878863623 wait complete. 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 Processor -- start 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.136+0000 7f67b8f37640 1 -- start start 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b8f37640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b41021a0 0x7f67b4101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b8f37640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 0x7f67b4100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b8f37640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4107400 0x7f67b4100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b8f37640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b410d360 con 0x7f67b4104ad0 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b8f37640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b410d1e0 con 0x7f67b4107400 2024-03-28T02:09:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b8f37640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b410d4e0 con 0x7f67b41021a0 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b1d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 0x7f67b4100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b1d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 0x7f67b4100150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39722/0 (socket says 172.21.15.106:39722) 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b1d9b640 1 -- 172.21.15.106:0/1001491869 learned_addr learned my addr 172.21.15.106:0/1001491869 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b259c640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b41021a0 0x7f67b4101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b2d9d640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4107400 0x7f67b4100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b1d9b640 1 -- 172.21.15.106:0/1001491869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b41021a0 msgr2=0x7f67b4101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b1d9b640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b41021a0 0x7f67b4101b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b1d9b640 1 -- 172.21.15.106:0/1001491869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4107400 msgr2=0x7f67b4100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b1d9b640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4107400 0x7f67b4100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b1d9b640 1 -- 172.21.15.106:0/1001491869 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b4100f20 con 0x7f67b4104ad0 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b1d9b640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 0x7f67b4100150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f67a4000c00 tx=0x7f67a402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67a37fe640 1 -- 172.21.15.106:0/1001491869 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a4043020 con 0x7f67b4104ad0 2024-03-28T02:09:03.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67a37fe640 1 -- 172.21.15.106:0/1001491869 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67a40054e0 con 0x7f67b4104ad0 2024-03-28T02:09:03.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b41a1b20 con 0x7f67b4104ad0 2024-03-28T02:09:03.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67a37fe640 1 -- 172.21.15.106:0/1001491869 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a4003ce0 con 0x7f67b4104ad0 2024-03-28T02:09:03.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.137+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b41a2090 con 0x7f67b4104ad0 2024-03-28T02:09:03.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.138+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b40fd620 con 0x7f67b4104ad0 2024-03-28T02:09:03.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.139+0000 7f67a37fe640 1 -- 172.21.15.106:0/1001491869 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f67a4046020 con 0x7f67b4104ad0 2024-03-28T02:09:03.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.140+0000 7f67a37fe640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f679c0a2880 0x7f679c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.140+0000 7f67b259c640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f679c0a2880 0x7f679c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.140+0000 7f67a37fe640 1 -- 172.21.15.106:0/1001491869 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f67a40f0d20 con 0x7f67b4104ad0 2024-03-28T02:09:03.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.140+0000 7f67b259c640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f679c0a2880 0x7f679c0a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f67a80095f0 tx=0x7f67a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:03.137 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.144+0000 7f67a37fe640 1 -- 172.21.15.106:0/1001491869 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f67a404d020 con 0x7f67b4104ad0 2024-03-28T02:09:03.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.321+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67b40622a0 con 0x7f67b4104ad0 2024-03-28T02:09:03.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.322+0000 7f67a37fe640 1 -- 172.21.15.106:0/1001491869 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7f67a40bdb80 con 0x7f67b4104ad0 2024-03-28T02:09:03.316 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:03.316 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:09:03.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f679c0a2880 msgr2=0x7f679c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f679c0a2880 0x7f679c0a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f67a80095f0 tx=0x7f67a8005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:03.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 msgr2=0x7f67b4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 0x7f67b4100150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f67a4000c00 tx=0x7f67a402c6c0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 shutdown_connections 2024-03-28T02:09:03.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f679c0a2880 0x7f679c0a4d40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4107400 0x7f67b4100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67b4104ad0 0x7f67b4100150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 --2- 172.21.15.106:0/1001491869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b41021a0 0x7f67b4101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.324+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 >> 172.21.15.106:0/1001491869 conn(0x7f67b40fbb30 msgr2=0x7f67b4102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:03.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.325+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 shutdown_connections 2024-03-28T02:09:03.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.325+0000 7f67b8f37640 1 -- 172.21.15.106:0/1001491869 wait complete. 2024-03-28T02:09:03.329 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T02:08:46.952662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T02:08:47.976273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T02:08:51.703592+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T02:09:03.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:09:03.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 -- 172.21.15.106:0/1919265321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c8103280 msgr2=0x7f34c8109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 --2- 172.21.15.106:0/1919265321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c8103280 0x7f34c8109b10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009be0 tx=0x7f34c002bef0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 -- 172.21.15.106:0/1919265321 shutdown_connections 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 --2- 172.21.15.106:0/1919265321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c8103280 0x7f34c8109b10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 --2- 172.21.15.106:0/1919265321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c81028c0 0x7f34c8102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 --2- 172.21.15.106:0/1919265321 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34c81016d0 0x7f34c8101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 -- 172.21.15.106:0/1919265321 >> 172.21.15.106:0/1919265321 conn(0x7f34c80fcc40 msgr2=0x7f34c80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 -- 172.21.15.106:0/1919265321 shutdown_connections 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 -- 172.21.15.106:0/1919265321 wait complete. 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 Processor -- start 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 -- start start 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.443+0000 7f34cef9e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34c81016d0 0x7f34c8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34cef9e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c81028c0 0x7f34c8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34cef9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c8103280 0x7f34c8073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34cef9e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c810d350 con 0x7f34c81028c0 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34cef9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c810d1d0 con 0x7f34c8103280 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34cef9e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c810d4d0 con 0x7f34c81016d0 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c81028c0 0x7f34c8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c81028c0 0x7f34c8073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39742/0 (socket says 172.21.15.106:39742) 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c7fff640 1 -- 172.21.15.106:0/2860122787 learned_addr learned my addr 172.21.15.106:0/2860122787 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34cd53b640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c8103280 0x7f34c8073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34ccd3a640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34c81016d0 0x7f34c8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c7fff640 1 -- 172.21.15.106:0/2860122787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34c81016d0 msgr2=0x7f34c8075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c7fff640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34c81016d0 0x7f34c8075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c7fff640 1 -- 172.21.15.106:0/2860122787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c8103280 msgr2=0x7f34c8073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c7fff640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c8103280 0x7f34c8073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c7fff640 1 -- 172.21.15.106:0/2860122787 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8074860 con 0x7f34c81028c0 2024-03-28T02:09:03.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c7fff640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c81028c0 0x7f34c8073a90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f34b400e9c0 tx=0x7f34b400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:03.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c5ffb640 1 -- 172.21.15.106:0/2860122787 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b400cde0 con 0x7f34c81028c0 2024-03-28T02:09:03.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c5ffb640 1 -- 172.21.15.106:0/2860122787 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34b40035f0 con 0x7f34c81028c0 2024-03-28T02:09:03.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c81a1a90 con 0x7f34c81028c0 2024-03-28T02:09:03.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.444+0000 7f34c5ffb640 1 -- 172.21.15.106:0/2860122787 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4010690 con 0x7f34c81028c0 2024-03-28T02:09:03.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.445+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c81a1fe0 con 0x7f34c81028c0 2024-03-28T02:09:03.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.445+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c81089f0 con 0x7f34c81028c0 2024-03-28T02:09:03.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.447+0000 7f34c5ffb640 1 -- 172.21.15.106:0/2860122787 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f34b40037a0 con 0x7f34c81028c0 2024-03-28T02:09:03.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.448+0000 7f34c5ffb640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34b80a2800 0x7f34b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.448+0000 7f34ccd3a640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34b80a2800 0x7f34b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.448+0000 7f34c5ffb640 1 -- 172.21.15.106:0/2860122787 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f34b40cf0d0 con 0x7f34c81028c0 2024-03-28T02:09:03.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.448+0000 7f34ccd3a640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34b80a2800 0x7f34b80a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f34c8102720 tx=0x7f34bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:03.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.452+0000 7f34c5ffb640 1 -- 172.21.15.106:0/2860122787 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f34b409bf30 con 0x7f34c81028c0 2024-03-28T02:09:03.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:03 smithi106 ceph-mon[31965]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 345 B/s wr, 0 op/s 2024-03-28T02:09:03.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:03 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1998325276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:09:03.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:03 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1031565169' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T02:09:03.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:03 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1001491869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:03.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.636+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34c80622a0 con 0x7f34c81028c0 2024-03-28T02:09:03.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.636+0000 7f34c5ffb640 1 -- 172.21.15.106:0/2860122787 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34b409b830 con 0x7f34c81028c0 2024-03-28T02:09:03.630 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:09:03.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.638+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34b80a2800 msgr2=0x7f34b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.638+0000 7f34cef9e640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34b80a2800 0x7f34b80a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f34c8102720 tx=0x7f34bc005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:03.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.638+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c81028c0 msgr2=0x7f34c8073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.638+0000 7f34cef9e640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c81028c0 0x7f34c8073a90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f34b400e9c0 tx=0x7f34b400ee80 comp rx=0 tx=0).stop 2024-03-28T02:09:03.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.638+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 shutdown_connections 2024-03-28T02:09:03.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.638+0000 7f34cef9e640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f34b80a2800 0x7f34b80a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.638+0000 7f34cef9e640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c8103280 0x7f34c8073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.638+0000 7f34cef9e640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34c81028c0 0x7f34c8073a90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.638+0000 7f34cef9e640 1 --2- 172.21.15.106:0/2860122787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34c81016d0 0x7f34c8075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.639+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 >> 172.21.15.106:0/2860122787 conn(0x7f34c80fcc40 msgr2=0x7f34c8105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:03.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.639+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 shutdown_connections 2024-03-28T02:09:03.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.639+0000 7f34cef9e640 1 -- 172.21.15.106:0/2860122787 wait complete. 2024-03-28T02:09:03.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T02:09:03.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.758+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1039909044 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93dc1017d0 msgr2=0x7f93dc101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.758+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1039909044 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93dc1017d0 0x7f93dc101c50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009ca0 tx=0x7f93d002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.758+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1039909044 shutdown_connections 2024-03-28T02:09:03.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.758+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1039909044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93dc102190 0x7f93dc108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.758+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1039909044 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93dc1017d0 0x7f93dc101c50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.758+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1039909044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93dc1005e0 0x7f93dc100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.758+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1039909044 >> 172.21.15.106:0/1039909044 conn(0x7f93dc0fbb50 msgr2=0x7f93dc0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:03.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.758+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1039909044 shutdown_connections 2024-03-28T02:09:03.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.758+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1039909044 wait complete. 2024-03-28T02:09:03.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.759+0000 7f93e0ab2640 1 Processor -- start 2024-03-28T02:09:03.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.759+0000 7f93e0ab2640 1 -- start start 2024-03-28T02:09:03.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.759+0000 7f93e0ab2640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93dc1005e0 0x7f93dc197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.759+0000 7f93e0ab2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93dc1017d0 0x7f93dc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.759+0000 7f93e0ab2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93dc102190 0x7f93dc19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.759+0000 7f93e0ab2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc10c240 con 0x7f93dc102190 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.759+0000 7f93e0ab2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc10c0c0 con 0x7f93dc1017d0 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.759+0000 7f93e0ab2640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc10c3c0 con 0x7f93dc1005e0 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93dad9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93dc102190 0x7f93dc19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93dad9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93dc102190 0x7f93dc19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39762/0 (socket says 172.21.15.106:39762) 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93dad9d640 1 -- 172.21.15.106:0/1723225212 learned_addr learned my addr 172.21.15.106:0/1723225212 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93d9d9b640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93dc1017d0 0x7f93dc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93da59c640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93dc1005e0 0x7f93dc197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93d9d9b640 1 -- 172.21.15.106:0/1723225212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93dc1005e0 msgr2=0x7f93dc197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93d9d9b640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93dc1005e0 0x7f93dc197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93d9d9b640 1 -- 172.21.15.106:0/1723225212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93dc102190 msgr2=0x7f93dc19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93d9d9b640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93dc102190 0x7f93dc19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93d9d9b640 1 -- 172.21.15.106:0/1723225212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc19f6c0 con 0x7f93dc1017d0 2024-03-28T02:09:03.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93d9d9b640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93dc1017d0 0x7f93dc198030 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f93d0006010 tx=0x7f93d0032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:03.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93cb7fe640 1 -- 172.21.15.106:0/1723225212 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d0041020 con 0x7f93dc1017d0 2024-03-28T02:09:03.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc19f950 con 0x7f93dc1017d0 2024-03-28T02:09:03.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93cb7fe640 1 -- 172.21.15.106:0/1723225212 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93d0034cd0 con 0x7f93dc1017d0 2024-03-28T02:09:03.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93cb7fe640 1 -- 172.21.15.106:0/1723225212 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d0039040 con 0x7f93dc1017d0 2024-03-28T02:09:03.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.760+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc19fe40 con 0x7f93dc1017d0 2024-03-28T02:09:03.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.761+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93dc1a0070 con 0x7f93dc1017d0 2024-03-28T02:09:03.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.762+0000 7f93cb7fe640 1 -- 172.21.15.106:0/1723225212 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f93d0044020 con 0x7f93dc1017d0 2024-03-28T02:09:03.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.763+0000 7f93cb7fe640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f93c40a2800 0x7f93c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:03.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.763+0000 7f93cb7fe640 1 -- 172.21.15.106:0/1723225212 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f93d00f0dc0 con 0x7f93dc1017d0 2024-03-28T02:09:03.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.763+0000 7f93da59c640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f93c40a2800 0x7f93c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:03.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.763+0000 7f93da59c640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f93c40a2800 0x7f93c40a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f93d40095f0 tx=0x7f93d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:03.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.767+0000 7f93cb7fe640 1 -- 172.21.15.106:0/1723225212 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f93d004b020 con 0x7f93dc1017d0 2024-03-28T02:09:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:03 smithi178 ceph-mon[36521]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 345 B/s wr, 0 op/s 2024-03-28T02:09:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:03 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1998325276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:09:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:03 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1031565169' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T02:09:03.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:03 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1001491869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:03.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.948+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f93dc1a0240 con 0x7f93dc1017d0 2024-03-28T02:09:03.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.949+0000 7f93cb7fe640 1 -- 172.21.15.106:0/1723225212 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f93d00bdc50 con 0x7f93dc1017d0 2024-03-28T02:09:03.942 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:09:03.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f93c40a2800 msgr2=0x7f93c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f93c40a2800 0x7f93c40a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f93d40095f0 tx=0x7f93d4005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:03.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93dc1017d0 msgr2=0x7f93dc198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:03.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93dc1017d0 0x7f93dc198030 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f93d0006010 tx=0x7f93d0032f40 comp rx=0 tx=0).stop 2024-03-28T02:09:03.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 shutdown_connections 2024-03-28T02:09:03.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93dc102190 0x7f93dc19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f93c40a2800 0x7f93c40a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93dc1017d0 0x7f93dc198030 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 --2- 172.21.15.106:0/1723225212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93dc1005e0 0x7f93dc197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:03.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 >> 172.21.15.106:0/1723225212 conn(0x7f93dc0fbb50 msgr2=0x7f93dc069a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:03.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 shutdown_connections 2024-03-28T02:09:03.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:03.951+0000 7f93e0ab2640 1 -- 172.21.15.106:0/1723225212 wait complete. 2024-03-28T02:09:03.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T02:09:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:03 smithi153 ceph-mon[36228]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 345 B/s wr, 0 op/s 2024-03-28T02:09:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:03 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1998325276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:09:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:03 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1031565169' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T02:09:03.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:03 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1001491869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.084+0000 7fde8a875640 1 -- 172.21.15.106:0/2582228783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101790 msgr2=0x7fde84101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.084+0000 7fde8a875640 1 --2- 172.21.15.106:0/2582228783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101790 0x7fde84101c10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde74009860 tx=0x7fde7402bb70 comp rx=0 tx=0).stop 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 -- 172.21.15.106:0/2582228783 shutdown_connections 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 --2- 172.21.15.106:0/2582228783 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde84102150 0x7fde841089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 --2- 172.21.15.106:0/2582228783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101790 0x7fde84101c10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 --2- 172.21.15.106:0/2582228783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde841005a0 0x7fde841009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 -- 172.21.15.106:0/2582228783 >> 172.21.15.106:0/2582228783 conn(0x7fde840fbb30 msgr2=0x7fde840fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 -- 172.21.15.106:0/2582228783 shutdown_connections 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 -- 172.21.15.106:0/2582228783 wait complete. 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 Processor -- start 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 -- start start 2024-03-28T02:09:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde841005a0 0x7fde84069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde84101790 0x7fde8419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84102150 0x7fde8419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8410c230 con 0x7fde84101790 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8410c0b0 con 0x7fde84102150 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.085+0000 7fde8a875640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8410c3b0 con 0x7fde841005a0 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde837fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde84101790 0x7fde8419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde837fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde84101790 0x7fde8419c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39776/0 (socket says 172.21.15.106:39776) 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde837fe640 1 -- 172.21.15.106:0/485167466 learned_addr learned my addr 172.21.15.106:0/485167466 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde88e12640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84102150 0x7fde8419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde83fff640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde841005a0 0x7fde84069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde837fe640 1 -- 172.21.15.106:0/485167466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde841005a0 msgr2=0x7fde84069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde837fe640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde841005a0 0x7fde84069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde837fe640 1 -- 172.21.15.106:0/485167466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84102150 msgr2=0x7fde8419ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde837fe640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84102150 0x7fde8419ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde837fe640 1 -- 172.21.15.106:0/485167466 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8419f620 con 0x7fde84101790 2024-03-28T02:09:04.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde837fe640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde84101790 0x7fde8419c720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fde740349e0 tx=0x7fde740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:04.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde817fa640 1 -- 172.21.15.106:0/485167466 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74041030 con 0x7fde84101790 2024-03-28T02:09:04.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8419f8b0 con 0x7fde84101790 2024-03-28T02:09:04.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde817fa640 1 -- 172.21.15.106:0/485167466 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde740347b0 con 0x7fde84101790 2024-03-28T02:09:04.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde817fa640 1 -- 172.21.15.106:0/485167466 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74039040 con 0x7fde84101790 2024-03-28T02:09:04.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.086+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8419fda0 con 0x7fde84101790 2024-03-28T02:09:04.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.087+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde841078e0 con 0x7fde84101790 2024-03-28T02:09:04.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.088+0000 7fde817fa640 1 -- 172.21.15.106:0/485167466 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fde74044070 con 0x7fde84101790 2024-03-28T02:09:04.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.089+0000 7fde817fa640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.089+0000 7fde83fff640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.089+0000 7fde817fa640 1 -- 172.21.15.106:0/485167466 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fde740f0330 con 0x7fde84101790 2024-03-28T02:09:04.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.089+0000 7fde83fff640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fde780096c0 tx=0x7fde78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:04.086 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.093+0000 7fde817fa640 1 -- 172.21.15.106:0/485167466 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fde740bd190 con 0x7fde84101790 2024-03-28T02:09:04.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.271+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde840622a0 con 0x7fde84101790 2024-03-28T02:09:04.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.271+0000 7fde817fa640 1 -- 172.21.15.106:0/485167466 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde740bca90 con 0x7fde84101790 2024-03-28T02:09:04.265 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 64 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fde6c0a2800 msgr2=0x7fde6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fde780096c0 tx=0x7fde78009260 comp rx=0 tx=0).stop 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde84101790 msgr2=0x7fde8419c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde84101790 0x7fde8419c720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fde740349e0 tx=0x7fde740042f0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 shutdown_connections 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84102150 0x7fde8419ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde84101790 0x7fde8419c720 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 --2- 172.21.15.106:0/485167466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde841005a0 0x7fde84069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 >> 172.21.15.106:0/485167466 conn(0x7fde840fbb30 msgr2=0x7fde84104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:04.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 shutdown_connections 2024-03-28T02:09:04.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.273+0000 7fde8a875640 1 -- 172.21.15.106:0/485167466 wait complete. 2024-03-28T02:09:04.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T02:09:04.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.388+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3415259128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc102150 msgr2=0x7fd9bc1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.388+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3415259128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc102150 0x7fd9bc1089e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009ca0 tx=0x7fd9b402c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.388+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3415259128 shutdown_connections 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.388+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3415259128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc102150 0x7fd9bc1089e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.388+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3415259128 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9bc101790 0x7fd9bc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3415259128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc1005a0 0x7fd9bc1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3415259128 >> 172.21.15.106:0/3415259128 conn(0x7fd9bc0fbb30 msgr2=0x7fd9bc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3415259128 shutdown_connections 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3415259128 wait complete. 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 Processor -- start 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 -- start start 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9bc1005a0 0x7fd9bc104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc101790 0x7fd9bc102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc102150 0x7fd9bc1030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc10c230 con 0x7fd9bc101790 2024-03-28T02:09:04.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc10c0b0 con 0x7fd9bc102150 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.389+0000 7fd9c19c4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc10c3b0 con 0x7fd9bc1005a0 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9ba7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc101790 0x7fd9bc102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9ba7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc101790 0x7fd9bc102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39790/0 (socket says 172.21.15.106:39790) 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9ba7fc640 1 -- 172.21.15.106:0/3178940840 learned_addr learned my addr 172.21.15.106:0/3178940840 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9bb7fe640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc102150 0x7fd9bc1030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9baffd640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9bc1005a0 0x7fd9bc104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9ba7fc640 1 -- 172.21.15.106:0/3178940840 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9bc1005a0 msgr2=0x7fd9bc104530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9ba7fc640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9bc1005a0 0x7fd9bc104530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9ba7fc640 1 -- 172.21.15.106:0/3178940840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc102150 msgr2=0x7fd9bc1030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9ba7fc640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc102150 0x7fd9bc1030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9ba7fc640 1 -- 172.21.15.106:0/3178940840 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc103940 con 0x7fd9bc101790 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9bb7fe640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc102150 0x7fd9bc1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9ba7fc640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc101790 0x7fd9bc102b70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00c970 tx=0x7fd9ac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:04.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9c09c2640 1 -- 172.21.15.106:0/3178940840 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac007bb0 con 0x7fd9bc101790 2024-03-28T02:09:04.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9c09c2640 1 -- 172.21.15.106:0/3178940840 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9ac007d50 con 0x7fd9bc101790 2024-03-28T02:09:04.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc1a1b30 con 0x7fd9bc101790 2024-03-28T02:09:04.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9c09c2640 1 -- 172.21.15.106:0/3178940840 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac00f450 con 0x7fd9bc101790 2024-03-28T02:09:04.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.390+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc1a2030 con 0x7fd9bc101790 2024-03-28T02:09:04.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.391+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc1a2260 con 0x7fd9bc101790 2024-03-28T02:09:04.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.392+0000 7fd9c09c2640 1 -- 172.21.15.106:0/3178940840 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd9ac004fe0 con 0x7fd9bc101790 2024-03-28T02:09:04.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.393+0000 7fd9c09c2640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9a40a2800 0x7fd9a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.393+0000 7fd9c09c2640 1 -- 172.21.15.106:0/3178940840 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd9ac0cedf0 con 0x7fd9bc101790 2024-03-28T02:09:04.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.393+0000 7fd9baffd640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9a40a2800 0x7fd9a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.393+0000 7fd9baffd640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9a40a2800 0x7fd9a40a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc1015f0 tx=0x7fd9b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:04.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.397+0000 7fd9c09c2640 1 -- 172.21.15.106:0/3178940840 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd9ac014030 con 0x7fd9bc101790 2024-03-28T02:09:04.571 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:04 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2860122787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:09:04.571 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:04 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1723225212' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:09:04.571 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:04 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/485167466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:09:04.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.580+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd9bc1a2550 con 0x7fd9bc101790 2024-03-28T02:09:04.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.581+0000 7fd9c09c2640 1 -- 172.21.15.106:0/3178940840 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd9ac09b600 con 0x7fd9bc101790 2024-03-28T02:09:04.574 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:09:04.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9a40a2800 msgr2=0x7fd9a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9a40a2800 0x7fd9a40a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc1015f0 tx=0x7fd9b0005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc101790 msgr2=0x7fd9bc102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc101790 0x7fd9bc102b70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00c970 tx=0x7fd9ac00ce30 comp rx=0 tx=0).stop 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 shutdown_connections 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd9a40a2800 0x7fd9a40a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc102150 0x7fd9bc1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9bc101790 0x7fd9bc102b70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 --2- 172.21.15.106:0/3178940840 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9bc1005a0 0x7fd9bc104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 >> 172.21.15.106:0/3178940840 conn(0x7fd9bc0fbb30 msgr2=0x7fd9bc0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 shutdown_connections 2024-03-28T02:09:04.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.583+0000 7fd9c19c4640 1 -- 172.21.15.106:0/3178940840 wait complete. 2024-03-28T02:09:04.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T02:09:04.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.698+0000 7f5d386d1640 1 -- 172.21.15.106:0/3420611889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 msgr2=0x7f5d301009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.698+0000 7f5d386d1640 1 --2- 172.21.15.106:0/3420611889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 0x7f5d301009c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009900 tx=0x7f5d2402bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.698+0000 7f5d386d1640 1 -- 172.21.15.106:0/3420611889 shutdown_connections 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.698+0000 7f5d386d1640 1 --2- 172.21.15.106:0/3420611889 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d30102150 0x7f5d301089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.698+0000 7f5d386d1640 1 --2- 172.21.15.106:0/3420611889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d30101790 0x7f5d30101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.698+0000 7f5d386d1640 1 --2- 172.21.15.106:0/3420611889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 0x7f5d301009c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.698+0000 7f5d386d1640 1 -- 172.21.15.106:0/3420611889 >> 172.21.15.106:0/3420611889 conn(0x7f5d300fbb30 msgr2=0x7f5d300fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.698+0000 7f5d386d1640 1 -- 172.21.15.106:0/3420611889 shutdown_connections 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.698+0000 7f5d386d1640 1 -- 172.21.15.106:0/3420611889 wait complete. 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d386d1640 1 Processor -- start 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d386d1640 1 -- start start 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d386d1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 0x7f5d30069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d386d1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d30101790 0x7f5d3019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d386d1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d30102150 0x7f5d3019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d386d1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3010c270 con 0x7f5d301005a0 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d386d1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3010c0f0 con 0x7f5d30102150 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d386d1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3010c3f0 con 0x7f5d30101790 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d3646d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 0x7f5d30069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d3646d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 0x7f5d30069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39808/0 (socket says 172.21.15.106:39808) 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d3646d640 1 -- 172.21.15.106:0/2009943793 learned_addr learned my addr 172.21.15.106:0/2009943793 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d36c6e640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d30102150 0x7f5d3019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d3646d640 1 -- 172.21.15.106:0/2009943793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d30101790 msgr2=0x7f5d3019c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d3646d640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d30101790 0x7f5d3019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d3646d640 1 -- 172.21.15.106:0/2009943793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d30102150 msgr2=0x7f5d3019ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d3646d640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d30102150 0x7f5d3019ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d3646d640 1 -- 172.21.15.106:0/2009943793 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3019f6a0 con 0x7f5d301005a0 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.699+0000 7f5d3646d640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 0x7f5d30069390 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5d24000c00 tx=0x7f5d24005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.700+0000 7f5d237fe640 1 -- 172.21.15.106:0/2009943793 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24041070 con 0x7f5d301005a0 2024-03-28T02:09:04.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.700+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3019f930 con 0x7f5d301005a0 2024-03-28T02:09:04.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.700+0000 7f5d237fe640 1 -- 172.21.15.106:0/2009943793 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d240044d0 con 0x7f5d301005a0 2024-03-28T02:09:04.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.700+0000 7f5d237fe640 1 -- 172.21.15.106:0/2009943793 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24034440 con 0x7f5d301005a0 2024-03-28T02:09:04.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.700+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3019fe70 con 0x7f5d301005a0 2024-03-28T02:09:04.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.701+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d30068490 con 0x7f5d301005a0 2024-03-28T02:09:04.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.702+0000 7f5d237fe640 1 -- 172.21.15.106:0/2009943793 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5d24044070 con 0x7f5d301005a0 2024-03-28T02:09:04.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.702+0000 7f5d237fe640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5d1c0a2800 0x7f5d1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:04.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.702+0000 7f5d35c6c640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5d1c0a2800 0x7f5d1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:04.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.702+0000 7f5d237fe640 1 -- 172.21.15.106:0/2009943793 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5d240f08d0 con 0x7f5d301005a0 2024-03-28T02:09:04.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.703+0000 7f5d35c6c640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5d1c0a2800 0x7f5d1c0a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0058c0 tx=0x7f5d2c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:04.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.707+0000 7f5d237fe640 1 -- 172.21.15.106:0/2009943793 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5d240bd730 con 0x7f5d301005a0 2024-03-28T02:09:04.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:04 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2860122787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:09:04.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:04 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1723225212' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:09:04.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:04 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/485167466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:09:04.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:04 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2860122787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:09:04.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:04 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1723225212' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:09:04.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:04 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/485167466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:09:04.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.894+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5d30107940 con 0x7f5d301005a0 2024-03-28T02:09:04.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.895+0000 7f5d237fe640 1 -- 172.21.15.106:0/2009943793 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+596 (secure 0 0 0) 0x7f5d240bd030 con 0x7f5d301005a0 2024-03-28T02:09:04.888 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/11266040 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:04.888 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/2884389364 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:04.888 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/4050392782 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:04.888 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6800/1510653090 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:04.888 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/103827077 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:04.888 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6801/305853075 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:04.888 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/256234961 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:04.888 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6800/305853075 2024-03-29T02:01:42.267941+0000 2024-03-28T02:09:04.889 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6801/1510653090 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:04.889 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/2759478196 2024-03-29T02:01:15.853032+0000 2024-03-28T02:09:04.889 INFO:teuthology.orchestra.run.smithi106.stderr:listed 10 entries 2024-03-28T02:09:04.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5d1c0a2800 msgr2=0x7f5d1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5d1c0a2800 0x7f5d1c0a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0058c0 tx=0x7f5d2c005850 comp rx=0 tx=0).stop 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 msgr2=0x7f5d30069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 0x7f5d30069390 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5d24000c00 tx=0x7f5d24005d50 comp rx=0 tx=0).stop 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 shutdown_connections 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5d1c0a2800 0x7f5d1c0a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d30102150 0x7f5d3019ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d30101790 0x7f5d3019c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 --2- 172.21.15.106:0/2009943793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d301005a0 0x7f5d30069390 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 >> 172.21.15.106:0/2009943793 conn(0x7f5d300fbb30 msgr2=0x7f5d30104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 shutdown_connections 2024-03-28T02:09:04.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:04.897+0000 7f5d386d1640 1 -- 172.21.15.106:0/2009943793 wait complete. 2024-03-28T02:09:04.901 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi106.front.sepia.ceph.com... 2024-03-28T02:09:04.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:09:04.901 DEBUG:teuthology.orchestra.run.smithi106:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-28T02:09:04.926 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi106.stderr:ceph-fuse[76246]: fuse finished with error 0 and tester_r 0 2024-03-28T02:09:04.928 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T02:09:04.984 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:09:04.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:09:04.984 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-28T02:09:05.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:09:05.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.153+0000 7feedb80b640 1 -- 172.21.15.106:0/4117464754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41016f0 msgr2=0x7feed4101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.153+0000 7feedb80b640 1 --2- 172.21.15.106:0/4117464754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41016f0 0x7feed4101b10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feecc009ca0 tx=0x7feecc02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.153+0000 7feedb80b640 1 -- 172.21.15.106:0/4117464754 shutdown_connections 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.153+0000 7feedb80b640 1 --2- 172.21.15.106:0/4117464754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feed41032a0 0x7feed4109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.153+0000 7feedb80b640 1 --2- 172.21.15.106:0/4117464754 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feed41028e0 0x7feed4102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.153+0000 7feedb80b640 1 --2- 172.21.15.106:0/4117464754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41016f0 0x7feed4101b10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.153+0000 7feedb80b640 1 -- 172.21.15.106:0/4117464754 >> 172.21.15.106:0/4117464754 conn(0x7feed40fcc40 msgr2=0x7feed40ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.153+0000 7feedb80b640 1 -- 172.21.15.106:0/4117464754 shutdown_connections 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feedb80b640 1 -- 172.21.15.106:0/4117464754 wait complete. 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feedb80b640 1 Processor -- start 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feedb80b640 1 -- start start 2024-03-28T02:09:05.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feedb80b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feed41016f0 0x7feed4075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feedb80b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feed41028e0 0x7feed4073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feedb80b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41032a0 0x7feed4073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feedb80b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed410d360 con 0x7feed41032a0 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feedb80b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed410d1e0 con 0x7feed41016f0 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feedb80b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed410d4e0 con 0x7feed41028e0 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feed9da8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41032a0 0x7feed4073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feed9da8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41032a0 0x7feed4073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39824/0 (socket says 172.21.15.106:39824) 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feed9da8640 1 -- 172.21.15.106:0/3830307613 learned_addr learned my addr 172.21.15.106:0/3830307613 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feed95a7640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feed41016f0 0x7feed4075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.154+0000 7feed8da6640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feed41028e0 0x7feed4073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feed9da8640 1 -- 172.21.15.106:0/3830307613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feed41028e0 msgr2=0x7feed4073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feed9da8640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feed41028e0 0x7feed4073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feed9da8640 1 -- 172.21.15.106:0/3830307613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feed41016f0 msgr2=0x7feed4075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feed9da8640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feed41016f0 0x7feed4075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feed9da8640 1 -- 172.21.15.106:0/3830307613 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed4074860 con 0x7feed41032a0 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feed95a7640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feed41016f0 0x7feed4075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feed9da8640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41032a0 0x7feed4073fd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feed000e970 tx=0x7feed000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feeca7fc640 1 -- 172.21.15.106:0/3830307613 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed000ccc0 con 0x7feed41032a0 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feeca7fc640 1 -- 172.21.15.106:0/3830307613 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feed000ce60 con 0x7feed41032a0 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed41a1b60 con 0x7feed41032a0 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feeca7fc640 1 -- 172.21.15.106:0/3830307613 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0010490 con 0x7feed41032a0 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.155+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed41a20b0 con 0x7feed41032a0 2024-03-28T02:09:05.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.156+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed4108a90 con 0x7feed41032a0 2024-03-28T02:09:05.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.157+0000 7feeca7fc640 1 -- 172.21.15.106:0/3830307613 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7feed0005020 con 0x7feed41032a0 2024-03-28T02:09:05.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.158+0000 7feeca7fc640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feec00a2880 0x7feec00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.158+0000 7feed95a7640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feec00a2880 0x7feec00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.158+0000 7feeca7fc640 1 -- 172.21.15.106:0/3830307613 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7feed00cf0c0 con 0x7feed41032a0 2024-03-28T02:09:05.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.158+0000 7feed95a7640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feec00a2880 0x7feec00a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feecc02c700 tx=0x7feecc032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:05.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.162+0000 7feeca7fc640 1 -- 172.21.15.106:0/3830307613 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feed009bf20 con 0x7feed41032a0 2024-03-28T02:09:05.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.341+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feed4062340 con 0x7feed41032a0 2024-03-28T02:09:05.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.343+0000 7feeca7fc640 1 -- 172.21.15.106:0/3830307613 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7feed009b820 con 0x7feed41032a0 2024-03-28T02:09:05.336 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:05.337 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feec00a2880 msgr2=0x7feec00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feec00a2880 0x7feec00a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feecc02c700 tx=0x7feecc032590 comp rx=0 tx=0).stop 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41032a0 msgr2=0x7feed4073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41032a0 0x7feed4073fd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feed000e970 tx=0x7feed000ed70 comp rx=0 tx=0).stop 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 shutdown_connections 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feed41032a0 0x7feed4073fd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feed41028e0 0x7feed4073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feec00a2880 0x7feec00a4d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 --2- 172.21.15.106:0/3830307613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feed41016f0 0x7feed4075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 >> 172.21.15.106:0/3830307613 conn(0x7feed40fcc40 msgr2=0x7feed4105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 shutdown_connections 2024-03-28T02:09:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.345+0000 7feedb80b640 1 -- 172.21.15.106:0/3830307613 wait complete. 2024-03-28T02:09:05.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.460+0000 7f714d597640 1 -- 172.21.15.106:0/366113851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71481028c0 msgr2=0x7f7148102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.460+0000 7f714d597640 1 --2- 172.21.15.106:0/366113851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71481028c0 0x7f7148102d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7130009900 tx=0x7f713002bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.460+0000 7f714d597640 1 -- 172.21.15.106:0/366113851 shutdown_connections 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.460+0000 7f714d597640 1 --2- 172.21.15.106:0/366113851 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7148103280 0x7f7148109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.460+0000 7f714d597640 1 --2- 172.21.15.106:0/366113851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71481028c0 0x7f7148102d40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.460+0000 7f714d597640 1 --2- 172.21.15.106:0/366113851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71481016d0 0x7f7148101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.460+0000 7f714d597640 1 -- 172.21.15.106:0/366113851 >> 172.21.15.106:0/366113851 conn(0x7f71480fcc40 msgr2=0x7f71480ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.460+0000 7f714d597640 1 -- 172.21.15.106:0/366113851 shutdown_connections 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.460+0000 7f714d597640 1 -- 172.21.15.106:0/366113851 wait complete. 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f714d597640 1 Processor -- start 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f714d597640 1 -- start start 2024-03-28T02:09:05.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f714d597640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71481016d0 0x7f71481979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f714d597640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71481028c0 0x7f7148197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f714d597640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7148103280 0x7f714819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f714d597640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714810d330 con 0x7f7148103280 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f714d597640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714810d1b0 con 0x7f71481028c0 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f714d597640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714810d4b0 con 0x7f71481016d0 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f71477fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7148103280 0x7f714819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f71477fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7148103280 0x7f714819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39842/0 (socket says 172.21.15.106:39842) 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f71477fe640 1 -- 172.21.15.106:0/669508876 learned_addr learned my addr 172.21.15.106:0/669508876 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f713ffff640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71481028c0 0x7f7148197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f7146ffd640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71481016d0 0x7f71481979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f71477fe640 1 -- 172.21.15.106:0/669508876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71481016d0 msgr2=0x7f71481979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f71477fe640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71481016d0 0x7f71481979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f71477fe640 1 -- 172.21.15.106:0/669508876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71481028c0 msgr2=0x7f7148197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f71477fe640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71481028c0 0x7f7148197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.461+0000 7f71477fe640 1 -- 172.21.15.106:0/669508876 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714819f600 con 0x7f7148103280 2024-03-28T02:09:05.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.462+0000 7f71477fe640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7148103280 0x7f714819ef90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f714000cce0 tx=0x7f714000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:05.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.462+0000 7f7144ff9640 1 -- 172.21.15.106:0/669508876 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714000f450 con 0x7f7148103280 2024-03-28T02:09:05.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.462+0000 7f7144ff9640 1 -- 172.21.15.106:0/669508876 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f714000fad0 con 0x7f7148103280 2024-03-28T02:09:05.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.462+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714819f8f0 con 0x7f7148103280 2024-03-28T02:09:05.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.462+0000 7f7144ff9640 1 -- 172.21.15.106:0/669508876 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7140018c10 con 0x7f7148103280 2024-03-28T02:09:05.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.462+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714819fdc0 con 0x7f7148103280 2024-03-28T02:09:05.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.463+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7148108990 con 0x7f7148103280 2024-03-28T02:09:05.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.464+0000 7f7144ff9640 1 -- 172.21.15.106:0/669508876 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f714000f5f0 con 0x7f7148103280 2024-03-28T02:09:05.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.465+0000 7f7144ff9640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f71340a2800 0x7f71340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.465+0000 7f7146ffd640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f71340a2800 0x7f71340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.465+0000 7f7144ff9640 1 -- 172.21.15.106:0/669508876 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f71400cf2f0 con 0x7f7148103280 2024-03-28T02:09:05.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.465+0000 7f7146ffd640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f71340a2800 0x7f71340a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f71380095f0 tx=0x7f7138005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:05.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.469+0000 7f7144ff9640 1 -- 172.21.15.106:0/669508876 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7140014030 con 0x7f7148103280 2024-03-28T02:09:05.641 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.648+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71480622a0 con 0x7f7148103280 2024-03-28T02:09:05.642 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.649+0000 7f7144ff9640 1 -- 172.21.15.106:0/669508876 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7f714009c150 con 0x7f7148103280 2024-03-28T02:09:05.643 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:05.643 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:09:05.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f71340a2800 msgr2=0x7f71340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f71340a2800 0x7f71340a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f71380095f0 tx=0x7f7138005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:05.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7148103280 msgr2=0x7f714819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7148103280 0x7f714819ef90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f714000cce0 tx=0x7f714000a620 comp rx=0 tx=0).stop 2024-03-28T02:09:05.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 shutdown_connections 2024-03-28T02:09:05.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7148103280 0x7f714819ef90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f71340a2800 0x7f71340a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71481028c0 0x7f7148197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 --2- 172.21.15.106:0/669508876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71481016d0 0x7f71481979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 >> 172.21.15.106:0/669508876 conn(0x7f71480fcc40 msgr2=0x7f7148105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:05.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 shutdown_connections 2024-03-28T02:09:05.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.651+0000 7f714d597640 1 -- 172.21.15.106:0/669508876 wait complete. 2024-03-28T02:09:05.654 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:05 smithi106 ceph-mon[31965]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T02:09:05.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:05 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3178940840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:09:05.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:05 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2009943793' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T02:09:05.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:05 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3830307613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:05.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T02:09:05.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.767+0000 7efc2fa6e640 1 -- 172.21.15.106:0/682255840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 msgr2=0x7efc281089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.767+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/682255840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 0x7efc281089e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efc20009ca0 tx=0x7efc2002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.767+0000 7efc2fa6e640 1 -- 172.21.15.106:0/682255840 shutdown_connections 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.767+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/682255840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 0x7efc281089e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.767+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/682255840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc28101790 0x7efc28101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.767+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/682255840 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc281005a0 0x7efc281009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.767+0000 7efc2fa6e640 1 -- 172.21.15.106:0/682255840 >> 172.21.15.106:0/682255840 conn(0x7efc280fbb30 msgr2=0x7efc280fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.767+0000 7efc2fa6e640 1 -- 172.21.15.106:0/682255840 shutdown_connections 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.767+0000 7efc2fa6e640 1 -- 172.21.15.106:0/682255840 wait complete. 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2fa6e640 1 Processor -- start 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2fa6e640 1 -- start start 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2fa6e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc281005a0 0x7efc2806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2fa6e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc28101790 0x7efc2819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2fa6e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 0x7efc2819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2fa6e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2810c290 con 0x7efc28102150 2024-03-28T02:09:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2fa6e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2810c110 con 0x7efc281005a0 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2fa6e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2810c410 con 0x7efc28101790 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2e00b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 0x7efc2819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2e00b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 0x7efc2819edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39856/0 (socket says 172.21.15.106:39856) 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2e00b640 1 -- 172.21.15.106:0/2628881909 learned_addr learned my addr 172.21.15.106:0/2628881909 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2d80a640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc281005a0 0x7efc2806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2d009640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc28101790 0x7efc2819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2e00b640 1 -- 172.21.15.106:0/2628881909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc28101790 msgr2=0x7efc2819c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2e00b640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc28101790 0x7efc2819c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2e00b640 1 -- 172.21.15.106:0/2628881909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc281005a0 msgr2=0x7efc2806a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2e00b640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc281005a0 0x7efc2806a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2e00b640 1 -- 172.21.15.106:0/2628881909 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2819f6c0 con 0x7efc28102150 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2e00b640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 0x7efc2819edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efc20032530 tx=0x7efc20032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:05.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc1effd640 1 -- 172.21.15.106:0/2628881909 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc20041020 con 0x7efc28102150 2024-03-28T02:09:05.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc1effd640 1 -- 172.21.15.106:0/2628881909 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc20034cd0 con 0x7efc28102150 2024-03-28T02:09:05.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2819f8f0 con 0x7efc28102150 2024-03-28T02:09:05.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.768+0000 7efc1effd640 1 -- 172.21.15.106:0/2628881909 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc20039040 con 0x7efc28102150 2024-03-28T02:09:05.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.769+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2819fdb0 con 0x7efc28102150 2024-03-28T02:09:05.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.770+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc28107860 con 0x7efc28102150 2024-03-28T02:09:05.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.771+0000 7efc1effd640 1 -- 172.21.15.106:0/2628881909 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efc20044020 con 0x7efc28102150 2024-03-28T02:09:05.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.771+0000 7efc1effd640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efc140a2800 0x7efc140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:05.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.771+0000 7efc2d80a640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efc140a2800 0x7efc140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:05.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.771+0000 7efc1effd640 1 -- 172.21.15.106:0/2628881909 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7efc200f0fe0 con 0x7efc28102150 2024-03-28T02:09:05.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.772+0000 7efc2d80a640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efc140a2800 0x7efc140a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efc281015f0 tx=0x7efc18009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:05.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.775+0000 7efc1effd640 1 -- 172.21.15.106:0/2628881909 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efc2004b020 con 0x7efc28102150 2024-03-28T02:09:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:05 smithi178 ceph-mon[36521]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T02:09:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:05 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3178940840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:09:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:05 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2009943793' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T02:09:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:05 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3830307613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:05.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:05.984+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7efc2802d270 con 0x7efc28102150 2024-03-28T02:09:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:05 smithi153 ceph-mon[36228]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T02:09:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:05 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3178940840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:09:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:05 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2009943793' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T02:09:05.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:05 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3830307613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:06.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.515+0000 7efc1effd640 1 -- 172.21.15.106:0/2628881909 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7efc200bded0 con 0x7efc28102150 2024-03-28T02:09:06.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efc140a2800 msgr2=0x7efc140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:06.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efc140a2800 0x7efc140a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efc281015f0 tx=0x7efc18009110 comp rx=0 tx=0).stop 2024-03-28T02:09:06.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 msgr2=0x7efc2819edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:06.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 0x7efc2819edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efc20032530 tx=0x7efc20032f40 comp rx=0 tx=0).stop 2024-03-28T02:09:06.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 shutdown_connections 2024-03-28T02:09:06.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc28102150 0x7efc2819edb0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc28101790 0x7efc2819c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7efc140a2800 0x7efc140a4cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 --2- 172.21.15.106:0/2628881909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc281005a0 0x7efc2806a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 >> 172.21.15.106:0/2628881909 conn(0x7efc280fbb30 msgr2=0x7efc28104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:06.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 shutdown_connections 2024-03-28T02:09:06.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.518+0000 7efc2fa6e640 1 -- 172.21.15.106:0/2628881909 wait complete. 2024-03-28T02:09:06.524 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-28T02:09:06.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.640+0000 7f13705c4640 1 -- 172.21.15.106:0/2472188006 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 msgr2=0x7f13681009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.640+0000 7f13705c4640 1 --2- 172.21.15.106:0/2472188006 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 0x7f13681009d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1358009ca0 tx=0x7f135802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.640+0000 7f13705c4640 1 -- 172.21.15.106:0/2472188006 shutdown_connections 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.640+0000 7f13705c4640 1 --2- 172.21.15.106:0/2472188006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1368102160 0x7f13681089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.640+0000 7f13705c4640 1 --2- 172.21.15.106:0/2472188006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681017a0 0x7f1368101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.640+0000 7f13705c4640 1 --2- 172.21.15.106:0/2472188006 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 0x7f13681009d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.640+0000 7f13705c4640 1 -- 172.21.15.106:0/2472188006 >> 172.21.15.106:0/2472188006 conn(0x7f13680fbb20 msgr2=0x7f13680fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.640+0000 7f13705c4640 1 -- 172.21.15.106:0/2472188006 shutdown_connections 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.640+0000 7f13705c4640 1 -- 172.21.15.106:0/2472188006 wait complete. 2024-03-28T02:09:06.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13705c4640 1 Processor -- start 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13705c4640 1 -- start start 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13705c4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 0x7f13681044f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13705c4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681017a0 0x7f1368102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13705c4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1368102160 0x7f13681030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13705c4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136810c220 con 0x7f13681005b0 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13705c4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136810c0a0 con 0x7f13681017a0 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13705c4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136810c3a0 con 0x7f1368102160 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136e360640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 0x7f13681044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136e360640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 0x7f13681044f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39876/0 (socket says 172.21.15.106:39876) 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136e360640 1 -- 172.21.15.106:0/1049406963 learned_addr learned my addr 172.21.15.106:0/1049406963 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:06.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136db5f640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681017a0 0x7f1368102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136eb61640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1368102160 0x7f13681030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136e360640 1 -- 172.21.15.106:0/1049406963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1368102160 msgr2=0x7f13681030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136e360640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1368102160 0x7f13681030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136e360640 1 -- 172.21.15.106:0/1049406963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681017a0 msgr2=0x7f1368102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136e360640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681017a0 0x7f1368102b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136e360640 1 -- 172.21.15.106:0/1049406963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1368103980 con 0x7f13681005b0 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136eb61640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1368102160 0x7f13681030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136db5f640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681017a0 0x7f1368102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f136e360640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 0x7f13681044f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f13580324c0 tx=0x7f1358032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13677fe640 1 -- 172.21.15.106:0/1049406963 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358041020 con 0x7f13681005b0 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.641+0000 7f13677fe640 1 -- 172.21.15.106:0/1049406963 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1358034cd0 con 0x7f13681005b0 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.642+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13681a1ae0 con 0x7f13681005b0 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.642+0000 7f13677fe640 1 -- 172.21.15.106:0/1049406963 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358039040 con 0x7f13681005b0 2024-03-28T02:09:06.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.642+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13681a1fa0 con 0x7f13681005b0 2024-03-28T02:09:06.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.643+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13681a2170 con 0x7f13681005b0 2024-03-28T02:09:06.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.644+0000 7f13677fe640 1 -- 172.21.15.106:0/1049406963 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1358044020 con 0x7f13681005b0 2024-03-28T02:09:06.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.644+0000 7f13677fe640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f13500a2800 0x7f13500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:06.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.644+0000 7f136db5f640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f13500a2800 0x7f13500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:06.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.644+0000 7f13677fe640 1 -- 172.21.15.106:0/1049406963 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f13580f1060 con 0x7f13681005b0 2024-03-28T02:09:06.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.645+0000 7f136db5f640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f13500a2800 0x7f13500a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1368104260 tx=0x7f135c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:06.641 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.648+0000 7f13677fe640 1 -- 172.21.15.106:0/1049406963 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f135804b020 con 0x7f13681005b0 2024-03-28T02:09:06.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:06 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/669508876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:06.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:06 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2628881909' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T02:09:06.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:06.870+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f13681a2500 con 0x7f13681005b0 2024-03-28T02:09:06.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:06 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/669508876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:06.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:06 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2628881909' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T02:09:06.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:06 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/669508876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:06.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:06 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2628881909' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T02:09:07.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.517+0000 7f13677fe640 1 -- 172.21.15.106:0/1049406963 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f13580bdf50 con 0x7f13681005b0 2024-03-28T02:09:07.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f13500a2800 msgr2=0x7f13500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f13500a2800 0x7f13500a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1368104260 tx=0x7f135c005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 msgr2=0x7f13681044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 0x7f13681044f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f13580324c0 tx=0x7f1358032f40 comp rx=0 tx=0).stop 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 shutdown_connections 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1368102160 0x7f13681030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f13500a2800 0x7f13500a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681017a0 0x7f1368102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 --2- 172.21.15.106:0/1049406963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13681005b0 0x7f13681044f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 >> 172.21.15.106:0/1049406963 conn(0x7f13680fbb20 msgr2=0x7f13680fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 shutdown_connections 2024-03-28T02:09:07.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.519+0000 7f13705c4640 1 -- 172.21.15.106:0/1049406963 wait complete. 2024-03-28T02:09:07.527 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T02:09:07.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.639+0000 7f0ac0473640 1 -- 172.21.15.106:0/3415487126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8103250 msgr2=0x7f0ab8109ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.639+0000 7f0ac0473640 1 --2- 172.21.15.106:0/3415487126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8103250 0x7f0ab8109ae0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009ca0 tx=0x7f0ab002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 -- 172.21.15.106:0/3415487126 shutdown_connections 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 --2- 172.21.15.106:0/3415487126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8103250 0x7f0ab8109ae0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 --2- 172.21.15.106:0/3415487126 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ab8102890 0x7f0ab8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 --2- 172.21.15.106:0/3415487126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ab81016a0 0x7f0ab8101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 -- 172.21.15.106:0/3415487126 >> 172.21.15.106:0/3415487126 conn(0x7f0ab80fcc10 msgr2=0x7f0ab80ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 -- 172.21.15.106:0/3415487126 shutdown_connections 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 -- 172.21.15.106:0/3415487126 wait complete. 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 Processor -- start 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 -- start start 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ab81016a0 0x7f0ab81979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8102890 0x7f0ab8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ab8103250 0x7f0ab819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab810d380 con 0x7f0ab8102890 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab810d200 con 0x7f0ab8103250 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0ac0473640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab810d500 con 0x7f0ab81016a0 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0abda0e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8102890 0x7f0ab8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0abda0e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8102890 0x7f0ab8197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35696/0 (socket says 172.21.15.106:35696) 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0abda0e640 1 -- 172.21.15.106:0/2061881131 learned_addr learned my addr 172.21.15.106:0/2061881131 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.640+0000 7f0abea10640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ab8103250 0x7f0ab819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0abe20f640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ab81016a0 0x7f0ab81979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0abda0e640 1 -- 172.21.15.106:0/2061881131 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ab81016a0 msgr2=0x7f0ab81979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0abda0e640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ab81016a0 0x7f0ab81979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0abda0e640 1 -- 172.21.15.106:0/2061881131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ab8103250 msgr2=0x7f0ab819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0abda0e640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ab8103250 0x7f0ab819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0abda0e640 1 -- 172.21.15.106:0/2061881131 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab819f610 con 0x7f0ab8102890 2024-03-28T02:09:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0abe20f640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ab81016a0 0x7f0ab81979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:09:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0abea10640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ab8103250 0x7f0ab819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0abda0e640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8102890 0x7f0ab8197f30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800b4a0 tx=0x7f0aa800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0aaf7fe640 1 -- 172.21.15.106:0/2061881131 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8014070 con 0x7f0ab8102890 2024-03-28T02:09:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab819f900 con 0x7f0ab8102890 2024-03-28T02:09:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0aaf7fe640 1 -- 172.21.15.106:0/2061881131 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0aa8010af0 con 0x7f0ab8102890 2024-03-28T02:09:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0aaf7fe640 1 -- 172.21.15.106:0/2061881131 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8017d10 con 0x7f0ab8102890 2024-03-28T02:09:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.641+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab819fe50 con 0x7f0ab8102890 2024-03-28T02:09:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.642+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab81089c0 con 0x7f0ab8102890 2024-03-28T02:09:07.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.643+0000 7f0aaf7fe640 1 -- 172.21.15.106:0/2061881131 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0aa8017070 con 0x7f0ab8102890 2024-03-28T02:09:07.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.644+0000 7f0aaf7fe640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0aa40a2800 0x7f0aa40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:07.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.644+0000 7f0abe20f640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0aa40a2800 0x7f0aa40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:07.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.644+0000 7f0aaf7fe640 1 -- 172.21.15.106:0/2061881131 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0aa80ce010 con 0x7f0ab8102890 2024-03-28T02:09:07.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.644+0000 7f0abe20f640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0aa40a2800 0x7f0aa40a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0ab81026f0 tx=0x7f0ab4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:07.642 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.649+0000 7f0aaf7fe640 1 -- 172.21.15.106:0/2061881131 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0aa809ae70 con 0x7f0ab8102890 2024-03-28T02:09:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:07 smithi106 ceph-mon[31965]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 258 B/s wr, 0 op/s 2024-03-28T02:09:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:07 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2628881909' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T02:09:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:07 smithi106 ceph-mon[31965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:09:07.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:07 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1049406963' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T02:09:07.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:07.861+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f0ab80622a0 con 0x7f0ab8102890 2024-03-28T02:09:07.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:07 smithi178 ceph-mon[36521]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 258 B/s wr, 0 op/s 2024-03-28T02:09:07.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:07 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2628881909' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T02:09:07.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:07 smithi178 ceph-mon[36521]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:09:07.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:07 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1049406963' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T02:09:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:07 smithi153 ceph-mon[36228]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 258 B/s wr, 0 op/s 2024-03-28T02:09:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:07 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2628881909' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T02:09:07.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:07 smithi153 ceph-mon[36228]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:09:07.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:07 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1049406963' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T02:09:08.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.242+0000 7f0aaf7fe640 1 -- 172.21.15.106:0/2061881131 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f0aa809a770 con 0x7f0ab8102890 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0aa40a2800 msgr2=0x7f0aa40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0aa40a2800 0x7f0aa40a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0ab81026f0 tx=0x7f0ab4005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8102890 msgr2=0x7f0ab8197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8102890 0x7f0ab8197f30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800b4a0 tx=0x7f0aa800b960 comp rx=0 tx=0).stop 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 shutdown_connections 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0aa40a2800 0x7f0aa40a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ab8103250 0x7f0ab819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ab8102890 0x7f0ab8197f30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 --2- 172.21.15.106:0/2061881131 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ab81016a0 0x7f0ab81979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 >> 172.21.15.106:0/2061881131 conn(0x7f0ab80fcc10 msgr2=0x7f0ab8105b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 shutdown_connections 2024-03-28T02:09:08.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.244+0000 7f0ac0473640 1 -- 172.21.15.106:0/2061881131 wait complete. 2024-03-28T02:09:08.254 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T02:09:08.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 -- 172.21.15.106:0/2236156716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c100f30 msgr2=0x7fa51c1013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:08.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 --2- 172.21.15.106:0/2236156716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c100f30 0x7fa51c1013b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009900 tx=0x7fa50c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 -- 172.21.15.106:0/2236156716 shutdown_connections 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 --2- 172.21.15.106:0/2236156716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c1018f0 0x7fa51c108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 --2- 172.21.15.106:0/2236156716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c100f30 0x7fa51c1013b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 --2- 172.21.15.106:0/2236156716 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa51c0ffd40 0x7fa51c100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 -- 172.21.15.106:0/2236156716 >> 172.21.15.106:0/2236156716 conn(0x7fa51c0fb2b0 msgr2=0x7fa51c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 -- 172.21.15.106:0/2236156716 shutdown_connections 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 -- 172.21.15.106:0/2236156716 wait complete. 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.366+0000 7fa523722640 1 Processor -- start 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa523722640 1 -- start start 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa523722640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c0ffd40 0x7fa51c196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa523722640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c100f30 0x7fa51c196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa523722640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa51c1018f0 0x7fa51c19dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:08.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa523722640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c10ba10 con 0x7fa51c0ffd40 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa523722640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c10b890 con 0x7fa51c100f30 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa523722640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c10bb90 con 0x7fa51c1018f0 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5214be640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c0ffd40 0x7fa51c196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5214be640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c0ffd40 0x7fa51c196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35710/0 (socket says 172.21.15.106:35710) 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5214be640 1 -- 172.21.15.106:0/2873624063 learned_addr learned my addr 172.21.15.106:0/2873624063 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa521cbf640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa51c1018f0 0x7fa51c19dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa520cbd640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c100f30 0x7fa51c196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5214be640 1 -- 172.21.15.106:0/2873624063 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa51c1018f0 msgr2=0x7fa51c19dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5214be640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa51c1018f0 0x7fa51c19dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5214be640 1 -- 172.21.15.106:0/2873624063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c100f30 msgr2=0x7fa51c196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5214be640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c100f30 0x7fa51c196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5214be640 1 -- 172.21.15.106:0/2873624063 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c19e530 con 0x7fa51c0ffd40 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5214be640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c0ffd40 0x7fa51c196910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa51800e9c0 tx=0x7fa51800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:08.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5127fc640 1 -- 172.21.15.106:0/2873624063 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51800cd80 con 0x7fa51c0ffd40 2024-03-28T02:09:08.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c19e820 con 0x7fa51c0ffd40 2024-03-28T02:09:08.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5127fc640 1 -- 172.21.15.106:0/2873624063 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5180035f0 con 0x7fa51c0ffd40 2024-03-28T02:09:08.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa5127fc640 1 -- 172.21.15.106:0/2873624063 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518010690 con 0x7fa51c0ffd40 2024-03-28T02:09:08.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.367+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c19ecf0 con 0x7fa51c0ffd40 2024-03-28T02:09:08.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.368+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51c107470 con 0x7fa51c0ffd40 2024-03-28T02:09:08.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.369+0000 7fa5127fc640 1 -- 172.21.15.106:0/2873624063 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa5180037a0 con 0x7fa51c0ffd40 2024-03-28T02:09:08.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.370+0000 7fa5127fc640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5080a2800 0x7fa5080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:08.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.370+0000 7fa520cbd640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5080a2800 0x7fa5080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:08.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.370+0000 7fa5127fc640 1 -- 172.21.15.106:0/2873624063 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa5180cf0d0 con 0x7fa51c0ffd40 2024-03-28T02:09:08.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.370+0000 7fa520cbd640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5080a2800 0x7fa5080a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa50c02c240 tx=0x7fa50c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:08.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.373+0000 7fa5127fc640 1 -- 172.21.15.106:0/2873624063 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa51809bf30 con 0x7fa51c0ffd40 2024-03-28T02:09:08.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:08.590+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fa51c02d070 con 0x7fa51c0ffd40 2024-03-28T02:09:08.732 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b[67179]: 2024-03-28T02:09:08.416+0000 7f3a59697700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T02:09:08.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T02:09:08.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1049406963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T02:09:08.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T02:09:08.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:09:08.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T02:09:08.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T02:09:08.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2061881131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T02:09:08.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2061881131' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T02:09:08.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] up:active 2024-03-28T02:09:08.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T02:09:08.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-28T02:09:08.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-03-28T02:09:08.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:08 smithi153 ceph-mon[36228]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T02:09:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T02:09:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1049406963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T02:09:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T02:09:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:09:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T02:09:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T02:09:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2061881131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T02:09:08.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2061881131' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T02:09:08.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] up:active 2024-03-28T02:09:08.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T02:09:08.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-28T02:09:08.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-03-28T02:09:08.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:08 smithi106 ceph-mon[31965]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1049406963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2061881131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2061881131' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] up:active 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-03-28T02:09:08.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:08 smithi178 ceph-mon[36521]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T02:09:09.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.257+0000 7fa5127fc640 1 -- 172.21.15.106:0/2873624063 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa51809b830 con 0x7fa51c0ffd40 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5080a2800 msgr2=0x7fa5080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5080a2800 0x7fa5080a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa50c02c240 tx=0x7fa50c02df90 comp rx=0 tx=0).stop 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c0ffd40 msgr2=0x7fa51c196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c0ffd40 0x7fa51c196910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa51800e9c0 tx=0x7fa51800ee80 comp rx=0 tx=0).stop 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 shutdown_connections 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa51c1018f0 0x7fa51c19dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5080a2800 0x7fa5080a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c100f30 0x7fa51c196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 --2- 172.21.15.106:0/2873624063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa51c0ffd40 0x7fa51c196910 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 >> 172.21.15.106:0/2873624063 conn(0x7fa51c0fb2b0 msgr2=0x7fa51c104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:09.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 shutdown_connections 2024-03-28T02:09:09.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.260+0000 7fa523722640 1 -- 172.21.15.106:0/2873624063 wait complete. 2024-03-28T02:09:09.269 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-28T02:09:09.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T02:09:09.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.385+0000 7fa4002d1640 1 -- 172.21.15.106:0/68852932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f8102160 msgr2=0x7fa3f81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.385+0000 7fa4002d1640 1 --2- 172.21.15.106:0/68852932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f8102160 0x7fa3f81089f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009ca0 tx=0x7fa3f002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.385+0000 7fa4002d1640 1 -- 172.21.15.106:0/68852932 shutdown_connections 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.385+0000 7fa4002d1640 1 --2- 172.21.15.106:0/68852932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f8102160 0x7fa3f81089f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.385+0000 7fa4002d1640 1 --2- 172.21.15.106:0/68852932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3f81017a0 0x7fa3f8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.385+0000 7fa4002d1640 1 --2- 172.21.15.106:0/68852932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f81005b0 0x7fa3f81009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.385+0000 7fa4002d1640 1 -- 172.21.15.106:0/68852932 >> 172.21.15.106:0/68852932 conn(0x7fa3f80fbb40 msgr2=0x7fa3f80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.385+0000 7fa4002d1640 1 -- 172.21.15.106:0/68852932 shutdown_connections 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.385+0000 7fa4002d1640 1 -- 172.21.15.106:0/68852932 wait complete. 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa4002d1640 1 Processor -- start 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa4002d1640 1 -- start start 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa4002d1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3f81005b0 0x7fa3f806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:09.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa4002d1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f81017a0 0x7fa3f819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa4002d1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f8102160 0x7fa3f819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa4002d1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f810c210 con 0x7fa3f81017a0 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa4002d1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f810c090 con 0x7fa3f8102160 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa4002d1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f810c390 con 0x7fa3f81005b0 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fd86c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f81017a0 0x7fa3f819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fd86c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f81017a0 0x7fa3f819c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35718/0 (socket says 172.21.15.106:35718) 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fd86c640 1 -- 172.21.15.106:0/1233649492 learned_addr learned my addr 172.21.15.106:0/1233649492 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fe06d640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3f81005b0 0x7fa3f806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fe86e640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f8102160 0x7fa3f819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fd86c640 1 -- 172.21.15.106:0/1233649492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3f81005b0 msgr2=0x7fa3f806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fd86c640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3f81005b0 0x7fa3f806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fd86c640 1 -- 172.21.15.106:0/1233649492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f8102160 msgr2=0x7fa3f819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fd86c640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f8102160 0x7fa3f819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fd86c640 1 -- 172.21.15.106:0/1233649492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f819f600 con 0x7fa3f81017a0 2024-03-28T02:09:09.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3fd86c640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f81017a0 0x7fa3f819c6d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400e9f0 tx=0x7fa3f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:09.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3ef7fe640 1 -- 172.21.15.106:0/1233649492 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f400cd80 con 0x7fa3f81017a0 2024-03-28T02:09:09.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3ef7fe640 1 -- 172.21.15.106:0/1233649492 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3f4010910 con 0x7fa3f81017a0 2024-03-28T02:09:09.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f819f890 con 0x7fa3f81017a0 2024-03-28T02:09:09.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.386+0000 7fa3ef7fe640 1 -- 172.21.15.106:0/1233649492 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4018990 con 0x7fa3f81017a0 2024-03-28T02:09:09.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.387+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f819fe30 con 0x7fa3f81017a0 2024-03-28T02:09:09.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.387+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f81078d0 con 0x7fa3f81017a0 2024-03-28T02:09:09.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.388+0000 7fa3ef7fe640 1 -- 172.21.15.106:0/1233649492 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa3f4010ac0 con 0x7fa3f81017a0 2024-03-28T02:09:09.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.389+0000 7fa3ef7fe640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa3e40a2880 0x7fa3e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:09.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.389+0000 7fa3fe06d640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa3e40a2880 0x7fa3e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:09.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.389+0000 7fa3ef7fe640 1 -- 172.21.15.106:0/1233649492 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa3f40cef50 con 0x7fa3f81017a0 2024-03-28T02:09:09.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.389+0000 7fa3fe06d640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa3e40a2880 0x7fa3e40a4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8101600 tx=0x7fa3e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:09.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.393+0000 7fa3ef7fe640 1 -- 172.21.15.106:0/1233649492 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa3f409b760 con 0x7fa3f81017a0 2024-03-28T02:09:09.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-mon[31965]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-28T02:09:09.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2873624063' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T02:09:09.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2873624063' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T02:09:09.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] up:active 2024-03-28T02:09:09.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-mon[31965]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T02:09:09.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-mon[31965]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T02:09:09.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-mon[31965]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-03-28T02:09:09.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-mon[31965]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-03-28T02:09:09.594 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.601+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3f8061ae0 con 0x7fa3e40a2880 2024-03-28T02:09:09.614 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.621+0000 7fa3ef7fe640 1 -- 172.21.15.106:0/1233649492 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa3f8061ae0 con 0x7fa3e40a2880 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa3e40a2880 msgr2=0x7fa3e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa3e40a2880 0x7fa3e40a4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8101600 tx=0x7fa3e8005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f81017a0 msgr2=0x7fa3f819c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f81017a0 0x7fa3f819c6d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400e9f0 tx=0x7fa3f400eeb0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 shutdown_connections 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa3e40a2880 0x7fa3e40a4d40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f8102160 0x7fa3f819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3f81017a0 0x7fa3f819c6d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 --2- 172.21.15.106:0/1233649492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3f81005b0 0x7fa3f806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 >> 172.21.15.106:0/1233649492 conn(0x7fa3f80fbb40 msgr2=0x7fa3f8104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 shutdown_connections 2024-03-28T02:09:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.623+0000 7fa4002d1640 1 -- 172.21.15.106:0/1233649492 wait complete. 2024-03-28T02:09:09.628 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.741+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1131769104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41028a0 msgr2=0x7f5dd4102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.741+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1131769104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41028a0 0x7f5dd4102d20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009900 tx=0x7f5dc402bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.741+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1131769104 shutdown_connections 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.741+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1131769104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dd4103260 0x7f5dd4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.741+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1131769104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41028a0 0x7f5dd4102d20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.741+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1131769104 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd41016b0 0x7f5dd4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.741+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1131769104 >> 172.21.15.106:0/1131769104 conn(0x7f5dd40fcc40 msgr2=0x7f5dd40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.741+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1131769104 shutdown_connections 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.741+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1131769104 wait complete. 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddc6b2640 1 Processor -- start 2024-03-28T02:09:09.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddc6b2640 1 -- start start 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddc6b2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41016b0 0x7f5dd41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddc6b2640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd41028a0 0x7f5dd4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddc6b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dd4103260 0x7f5dd419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddc6b2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd410d330 con 0x7f5dd41016b0 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddc6b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd410d1b0 con 0x7f5dd4103260 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddc6b2640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd410d4b0 con 0x7f5dd41028a0 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dda44e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41016b0 0x7f5dd41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dda44e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41016b0 0x7f5dd41979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35746/0 (socket says 172.21.15.106:35746) 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dda44e640 1 -- 172.21.15.106:0/1932369272 learned_addr learned my addr 172.21.15.106:0/1932369272 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dd9c4d640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd41028a0 0x7f5dd4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddac4f640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dd4103260 0x7f5dd419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dda44e640 1 -- 172.21.15.106:0/1932369272 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd41028a0 msgr2=0x7f5dd4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dda44e640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd41028a0 0x7f5dd4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dda44e640 1 -- 172.21.15.106:0/1932369272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dd4103260 msgr2=0x7f5dd419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dda44e640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dd4103260 0x7f5dd419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dda44e640 1 -- 172.21.15.106:0/1932369272 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd419f5f0 con 0x7f5dd41016b0 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5ddac4f640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dd4103260 0x7f5dd419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.742+0000 7f5dda44e640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41016b0 0x7f5dd41979d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000e9c0 tx=0x7f5dd000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.743+0000 7f5dcb7fe640 1 -- 172.21.15.106:0/1932369272 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd000cd80 con 0x7f5dd41016b0 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.743+0000 7f5dcb7fe640 1 -- 172.21.15.106:0/1932369272 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5dd00035f0 con 0x7f5dd41016b0 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.743+0000 7f5dcb7fe640 1 -- 172.21.15.106:0/1932369272 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0010690 con 0x7f5dd41016b0 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.743+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd419f8e0 con 0x7f5dd41016b0 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.743+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd419fe00 con 0x7f5dd41016b0 2024-03-28T02:09:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.744+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd4108aa0 con 0x7f5dd41016b0 2024-03-28T02:09:09.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.745+0000 7f5dcb7fe640 1 -- 172.21.15.106:0/1932369272 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5dd00037a0 con 0x7f5dd41016b0 2024-03-28T02:09:09.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.745+0000 7f5dcb7fe640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5dc00a2880 0x7f5dc00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:09.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.745+0000 7f5dd9c4d640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5dc00a2880 0x7f5dc00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:09.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.745+0000 7f5dcb7fe640 1 -- 172.21.15.106:0/1932369272 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5dd00cf020 con 0x7f5dd41016b0 2024-03-28T02:09:09.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.746+0000 7f5dd9c4d640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5dc00a2880 0x7f5dc00a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5dc402c7b0 tx=0x7f5dc4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:09.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.750+0000 7f5dcb7fe640 1 -- 172.21.15.106:0/1932369272 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5dd009be80 con 0x7f5dd41016b0 2024-03-28T02:09:09.833 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:09:09.606+0000 7efbbdc91700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:09:09.833 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:09:09.606+0000 7efbbdc91700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:09:09.833 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:09:09.606+0000 7efbbdc91700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:09:09.833 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:09:09.606+0000 7efbbdc91700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:09:09.833 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:09:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:09:09.606+0000 7efbbdc91700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:09:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:09 smithi178 ceph-mon[36521]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-28T02:09:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:09 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2873624063' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T02:09:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:09 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2873624063' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T02:09:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:09 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] up:active 2024-03-28T02:09:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:09 smithi178 ceph-mon[36521]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T02:09:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:09 smithi178 ceph-mon[36521]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T02:09:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:09 smithi178 ceph-mon[36521]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-03-28T02:09:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:09 smithi178 ceph-mon[36521]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-03-28T02:09:09.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.952+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f5dd4061ae0 con 0x7f5dc00a2880 2024-03-28T02:09:09.968 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.975+0000 7f5dcb7fe640 1 -- 172.21.15.106:0/1932369272 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5dd4061ae0 con 0x7f5dc00a2880 2024-03-28T02:09:09.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5dc00a2880 msgr2=0x7f5dc00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5dc00a2880 0x7f5dc00a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5dc402c7b0 tx=0x7f5dc4005fd0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41016b0 msgr2=0x7f5dd41979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41016b0 0x7f5dd41979d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000e9c0 tx=0x7f5dd000ee80 comp rx=0 tx=0).stop 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 shutdown_connections 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f5dc00a2880 0x7f5dc00a4d40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dd4103260 0x7f5dd419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd41028a0 0x7f5dd4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 --2- 172.21.15.106:0/1932369272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dd41016b0 0x7f5dd41979d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 >> 172.21.15.106:0/1932369272 conn(0x7f5dd40fcc40 msgr2=0x7f5dd4105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 shutdown_connections 2024-03-28T02:09:09.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:09.977+0000 7f5ddc6b2640 1 -- 172.21.15.106:0/1932369272 wait complete. 2024-03-28T02:09:09.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T02:09:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:09 smithi153 ceph-mon[36228]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-28T02:09:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:09 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2873624063' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T02:09:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:09 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2873624063' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T02:09:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:09 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] up:active 2024-03-28T02:09:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:09 smithi153 ceph-mon[36228]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T02:09:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:09 smithi153 ceph-mon[36228]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T02:09:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:09 smithi153 ceph-mon[36228]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-03-28T02:09:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:09 smithi153 ceph-mon[36228]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-03-28T02:09:10.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2174407058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01005b0 msgr2=0x7fa1e01009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2174407058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01005b0 0x7fa1e01009d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009900 tx=0x7fa1d002bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:10.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2174407058 shutdown_connections 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2174407058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e0102160 0x7fa1e01089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2174407058 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1e01017a0 0x7fa1e0101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2174407058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01005b0 0x7fa1e01009d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2174407058 >> 172.21.15.106:0/2174407058 conn(0x7fa1e00fbb40 msgr2=0x7fa1e00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2174407058 shutdown_connections 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2174407058 wait complete. 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 Processor -- start 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.095+0000 7fa1e56e9640 1 -- start start 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1e56e9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e01005b0 0x7fa1e01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1e56e9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01017a0 0x7fa1e0196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1e56e9640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1e0102160 0x7fa1e019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1e56e9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e010c200 con 0x7fa1e01017a0 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1e56e9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e010c080 con 0x7fa1e01005b0 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1e56e9640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e010c380 con 0x7fa1e0102160 2024-03-28T02:09:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1de7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01017a0 0x7fa1e0196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1de7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01017a0 0x7fa1e0196e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35764/0 (socket says 172.21.15.106:35764) 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1de7fc640 1 -- 172.21.15.106:0/2286575724 learned_addr learned my addr 172.21.15.106:0/2286575724 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1df7fe640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1e0102160 0x7fa1e019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1deffd640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e01005b0 0x7fa1e01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1de7fc640 1 -- 172.21.15.106:0/2286575724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1e0102160 msgr2=0x7fa1e019de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1de7fc640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1e0102160 0x7fa1e019de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1de7fc640 1 -- 172.21.15.106:0/2286575724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e01005b0 msgr2=0x7fa1e01968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1de7fc640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e01005b0 0x7fa1e01968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1de7fc640 1 -- 172.21.15.106:0/2286575724 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e019e4f0 con 0x7fa1e01017a0 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1de7fc640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01017a0 0x7fa1e0196e10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa1d400eb70 tx=0x7fa1d400b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1c7fff640 1 -- 172.21.15.106:0/2286575724 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d400be60 con 0x7fa1e01017a0 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e0069cd0 con 0x7fa1e01017a0 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1c7fff640 1 -- 172.21.15.106:0/2286575724 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1d40035f0 con 0x7fa1e01017a0 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e006a270 con 0x7fa1e01017a0 2024-03-28T02:09:10.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.096+0000 7fa1c7fff640 1 -- 172.21.15.106:0/2286575724 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4014680 con 0x7fa1e01017a0 2024-03-28T02:09:10.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.097+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e00688e0 con 0x7fa1e01017a0 2024-03-28T02:09:10.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.098+0000 7fa1c7fff640 1 -- 172.21.15.106:0/2286575724 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa1d40037a0 con 0x7fa1e01017a0 2024-03-28T02:09:10.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.099+0000 7fa1c7fff640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa1c80a2800 0x7fa1c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.099+0000 7fa1deffd640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa1c80a2800 0x7fa1c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.099+0000 7fa1c7fff640 1 -- 172.21.15.106:0/2286575724 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa1d40d09f0 con 0x7fa1e01017a0 2024-03-28T02:09:10.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.099+0000 7fa1deffd640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa1c80a2800 0x7fa1c80a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0000c00 tx=0x7fa1d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:10.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.103+0000 7fa1c7fff640 1 -- 172.21.15.106:0/2286575724 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa1d409d200 con 0x7fa1e01017a0 2024-03-28T02:09:10.301 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.308+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1e006a4d0 con 0x7fa1c80a2800 2024-03-28T02:09:10.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.318+0000 7fa1c7fff640 1 -- 172.21.15.106:0/2286575724 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fa1e006a4d0 con 0x7fa1c80a2800 2024-03-28T02:09:10.311 INFO:teuthology.orchestra.run.smithi106.stdout:/volumes/qa/sv_0/7c41974c-e142-429b-ae2a-8ba5808668e3 2024-03-28T02:09:10.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa1c80a2800 msgr2=0x7fa1c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa1c80a2800 0x7fa1c80a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0000c00 tx=0x7fa1d0005fd0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01017a0 msgr2=0x7fa1e0196e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01017a0 0x7fa1e0196e10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa1d400eb70 tx=0x7fa1d400b690 comp rx=0 tx=0).stop 2024-03-28T02:09:10.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 shutdown_connections 2024-03-28T02:09:10.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1e0102160 0x7fa1e019de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1e01017a0 0x7fa1e0196e10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa1c80a2800 0x7fa1c80a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 --2- 172.21.15.106:0/2286575724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e01005b0 0x7fa1e01968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 >> 172.21.15.106:0/2286575724 conn(0x7fa1e00fbb40 msgr2=0x7fa1e0104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:10.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 shutdown_connections 2024-03-28T02:09:10.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.320+0000 7fa1e56e9640 1 -- 172.21.15.106:0/2286575724 wait complete. 2024-03-28T02:09:10.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.438+0000 7fab359a9640 1 -- 172.21.15.106:0/2488828278 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab301005c0 msgr2=0x7fab301009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.438+0000 7fab359a9640 1 --2- 172.21.15.106:0/2488828278 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab301005c0 0x7fab301009e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fab28009900 tx=0x7fab2802bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 -- 172.21.15.106:0/2488828278 shutdown_connections 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 --2- 172.21.15.106:0/2488828278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab30102170 0x7fab30108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 --2- 172.21.15.106:0/2488828278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab301017b0 0x7fab30101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 --2- 172.21.15.106:0/2488828278 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab301005c0 0x7fab301009e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 -- 172.21.15.106:0/2488828278 >> 172.21.15.106:0/2488828278 conn(0x7fab300fbb50 msgr2=0x7fab300fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 -- 172.21.15.106:0/2488828278 shutdown_connections 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 -- 172.21.15.106:0/2488828278 wait complete. 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 Processor -- start 2024-03-28T02:09:10.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 -- start start 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab301005c0 0x7fab30197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab301017b0 0x7fab30197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.439+0000 7fab359a9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab30102170 0x7fab3019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab359a9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3010d360 con 0x7fab30102170 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab359a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3010d1e0 con 0x7fab301017b0 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab359a9640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3010d4e0 con 0x7fab301005c0 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab30102170 0x7fab3019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab30102170 0x7fab3019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35794/0 (socket says 172.21.15.106:35794) 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2f7fe640 1 -- 172.21.15.106:0/805497673 learned_addr learned my addr 172.21.15.106:0/805497673 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2e7fc640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab301017b0 0x7fab30197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2effd640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab301005c0 0x7fab30197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2f7fe640 1 -- 172.21.15.106:0/805497673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab301005c0 msgr2=0x7fab30197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2f7fe640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab301005c0 0x7fab30197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2f7fe640 1 -- 172.21.15.106:0/805497673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab301017b0 msgr2=0x7fab30197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2f7fe640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab301017b0 0x7fab30197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2f7fe640 1 -- 172.21.15.106:0/805497673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3019f660 con 0x7fab30102170 2024-03-28T02:09:10.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab2f7fe640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab30102170 0x7fab3019eff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fab2000eb10 tx=0x7fab2000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:10.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab349a7640 1 -- 172.21.15.106:0/805497673 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab20004b20 con 0x7fab30102170 2024-03-28T02:09:10.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab349a7640 1 -- 172.21.15.106:0/805497673 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab20004cc0 con 0x7fab30102170 2024-03-28T02:09:10.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3019f950 con 0x7fab30102170 2024-03-28T02:09:10.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab349a7640 1 -- 172.21.15.106:0/805497673 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab20010760 con 0x7fab30102170 2024-03-28T02:09:10.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.440+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3019fe50 con 0x7fab30102170 2024-03-28T02:09:10.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.441+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab301079b0 con 0x7fab30102170 2024-03-28T02:09:10.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.443+0000 7fab349a7640 1 -- 172.21.15.106:0/805497673 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fab20010900 con 0x7fab30102170 2024-03-28T02:09:10.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.443+0000 7fab349a7640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab180a2800 0x7fab180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.444+0000 7fab2effd640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab180a2800 0x7fab180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.444+0000 7fab349a7640 1 -- 172.21.15.106:0/805497673 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fab200cf440 con 0x7fab30102170 2024-03-28T02:09:10.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.444+0000 7fab2effd640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab180a2800 0x7fab180a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab28000c00 tx=0x7fab28005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:10.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.447+0000 7fab349a7640 1 -- 172.21.15.106:0/805497673 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab20053740 con 0x7fab30102170 2024-03-28T02:09:10.644 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.651+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fab30061ae0 con 0x7fab180a2800 2024-03-28T02:09:10.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.675+0000 7fab349a7640 1 -- 172.21.15.106:0/805497673 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fab30061ae0 con 0x7fab180a2800 2024-03-28T02:09:10.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab180a2800 msgr2=0x7fab180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab180a2800 0x7fab180a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab28000c00 tx=0x7fab28005fd0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab30102170 msgr2=0x7fab3019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab30102170 0x7fab3019eff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fab2000eb10 tx=0x7fab2000c2d0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 shutdown_connections 2024-03-28T02:09:10.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab30102170 0x7fab3019eff0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fab180a2800 0x7fab180a4cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab301017b0 0x7fab30197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 --2- 172.21.15.106:0/805497673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab301005c0 0x7fab30197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.677+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 >> 172.21.15.106:0/805497673 conn(0x7fab300fbb50 msgr2=0x7fab30104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:10.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.678+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 shutdown_connections 2024-03-28T02:09:10.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.678+0000 7fab359a9640 1 -- 172.21.15.106:0/805497673 wait complete. 2024-03-28T02:09:10.682 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.795+0000 7fd839cd1640 1 -- 172.21.15.106:0/3022028058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd834103280 msgr2=0x7fd834109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.795+0000 7fd839cd1640 1 --2- 172.21.15.106:0/3022028058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd834103280 0x7fd834109b10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009ca0 tx=0x7fd82c02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 -- 172.21.15.106:0/3022028058 shutdown_connections 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 --2- 172.21.15.106:0/3022028058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd834103280 0x7fd834109b10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 --2- 172.21.15.106:0/3022028058 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8341028c0 0x7fd834102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 --2- 172.21.15.106:0/3022028058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8341016d0 0x7fd834101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 -- 172.21.15.106:0/3022028058 >> 172.21.15.106:0/3022028058 conn(0x7fd8340fcc40 msgr2=0x7fd8340ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 -- 172.21.15.106:0/3022028058 shutdown_connections 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 -- 172.21.15.106:0/3022028058 wait complete. 2024-03-28T02:09:10.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 Processor -- start 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 -- start start 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8341016d0 0x7fd8341979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8341028c0 0x7fd834197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd834103280 0x7fd83419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83410d330 con 0x7fd8341028c0 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83410d1b0 con 0x7fd8341016d0 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.796+0000 7fd839cd1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83410d4b0 con 0x7fd834103280 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd832ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8341028c0 0x7fd834197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd8337fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8341016d0 0x7fd8341979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd832ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8341028c0 0x7fd834197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35808/0 (socket says 172.21.15.106:35808) 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd8337fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8341016d0 0x7fd8341979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:50970/0 (socket says 172.21.15.106:50970) 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd832ffd640 1 -- 172.21.15.106:0/1866262196 learned_addr learned my addr 172.21.15.106:0/1866262196 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd833fff640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd834103280 0x7fd83419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd832ffd640 1 -- 172.21.15.106:0/1866262196 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd834103280 msgr2=0x7fd83419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd832ffd640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd834103280 0x7fd83419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd832ffd640 1 -- 172.21.15.106:0/1866262196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8341016d0 msgr2=0x7fd8341979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd832ffd640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8341016d0 0x7fd8341979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd832ffd640 1 -- 172.21.15.106:0/1866262196 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83419f5c0 con 0x7fd8341028c0 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd8337fe640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8341016d0 0x7fd8341979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd832ffd640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8341028c0 0x7fd834197ee0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd82800e9f0 tx=0x7fd82800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd830ff9640 1 -- 172.21.15.106:0/1866262196 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82800cd80 con 0x7fd8341028c0 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83419f8b0 con 0x7fd8341028c0 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd830ff9640 1 -- 172.21.15.106:0/1866262196 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd828010910 con 0x7fd8341028c0 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd830ff9640 1 -- 172.21.15.106:0/1866262196 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8280188f0 con 0x7fd8341028c0 2024-03-28T02:09:10.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.797+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83419fe00 con 0x7fd8341028c0 2024-03-28T02:09:10.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.798+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8341089f0 con 0x7fd8341028c0 2024-03-28T02:09:10.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.799+0000 7fd830ff9640 1 -- 172.21.15.106:0/1866262196 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd828010ac0 con 0x7fd8341028c0 2024-03-28T02:09:10.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.800+0000 7fd830ff9640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd81c0a2800 0x7fd81c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:10.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.800+0000 7fd8337fe640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd81c0a2800 0x7fd81c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:10.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.800+0000 7fd830ff9640 1 -- 172.21.15.106:0/1866262196 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd8280cf050 con 0x7fd8341028c0 2024-03-28T02:09:10.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.800+0000 7fd8337fe640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd81c0a2800 0x7fd81c0a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd834102720 tx=0x7fd824005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:10.797 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:10.804+0000 7fd830ff9640 1 -- 172.21.15.106:0/1866262196 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd82809b860 con 0x7fd8341028c0 2024-03-28T02:09:11.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.008+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd834061ae0 con 0x7fd81c0a2800 2024-03-28T02:09:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.016+0000 7fd830ff9640 1 -- 172.21.15.106:0/1866262196 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fd834061ae0 con 0x7fd81c0a2800 2024-03-28T02:09:11.009 INFO:teuthology.orchestra.run.smithi106.stdout:/volumes/qa/sv_1/018493ee-3c1f-4528-b707-618afc45dfef 2024-03-28T02:09:11.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.017+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd81c0a2800 msgr2=0x7fd81c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd81c0a2800 0x7fd81c0a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd834102720 tx=0x7fd824005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:11.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8341028c0 msgr2=0x7fd834197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8341028c0 0x7fd834197ee0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd82800e9f0 tx=0x7fd82800eeb0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 shutdown_connections 2024-03-28T02:09:11.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd834103280 0x7fd83419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8341028c0 0x7fd834197ee0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd81c0a2800 0x7fd81c0a4cc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 --2- 172.21.15.106:0/1866262196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8341016d0 0x7fd8341979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 >> 172.21.15.106:0/1866262196 conn(0x7fd8340fcc40 msgr2=0x7fd834105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:11.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 shutdown_connections 2024-03-28T02:09:11.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.018+0000 7fd839cd1640 1 -- 172.21.15.106:0/1866262196 wait complete. 2024-03-28T02:09:11.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:09:11.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 -- 172.21.15.106:0/4196422929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 msgr2=0x7fcd78109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/4196422929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 0x7fcd78109bd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009cd0 tx=0x7fcd7402c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 -- 172.21.15.106:0/4196422929 shutdown_connections 2024-03-28T02:09:11.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/4196422929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 0x7fcd78109bd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/4196422929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78102980 0x7fcd78102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/4196422929 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd780ff9b0 0x7fcd780ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 -- 172.21.15.106:0/4196422929 >> 172.21.15.106:0/4196422929 conn(0x7fcd780fb2e0 msgr2=0x7fcd780fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 -- 172.21.15.106:0/4196422929 shutdown_connections 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 -- 172.21.15.106:0/4196422929 wait complete. 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 Processor -- start 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.130+0000 7fcd7ff62640 1 -- start start 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7ff62640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd780ff9b0 0x7fcd781979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7ff62640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78102980 0x7fcd78197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7ff62640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 0x7fcd7819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7ff62640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7810d360 con 0x7fcd78103340 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7ff62640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7810d1e0 con 0x7fcd78102980 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7ff62640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7810d4e0 con 0x7fcd780ff9b0 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7e4ff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 0x7fcd7819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7e4ff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 0x7fcd7819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35826/0 (socket says 172.21.15.106:35826) 2024-03-28T02:09:11.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7e4ff640 1 -- 172.21.15.106:0/2815885271 learned_addr learned my addr 172.21.15.106:0/2815885271 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7dcfe640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd780ff9b0 0x7fcd781979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7d4fd640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78102980 0x7fcd78197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7e4ff640 1 -- 172.21.15.106:0/2815885271 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd780ff9b0 msgr2=0x7fcd781979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7e4ff640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd780ff9b0 0x7fcd781979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7e4ff640 1 -- 172.21.15.106:0/2815885271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78102980 msgr2=0x7fcd78197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7e4ff640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78102980 0x7fcd78197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7e4ff640 1 -- 172.21.15.106:0/2815885271 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7819f5d0 con 0x7fcd78103340 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7e4ff640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 0x7fcd7819ef60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009ca0 tx=0x7fcd740346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd6effd640 1 -- 172.21.15.106:0/2815885271 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd74041020 con 0x7fcd78103340 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd6effd640 1 -- 172.21.15.106:0/2815885271 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd74032890 con 0x7fcd78103340 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd6effd640 1 -- 172.21.15.106:0/2815885271 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd74039040 con 0x7fcd78103340 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7819f860 con 0x7fcd78103340 2024-03-28T02:09:11.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.131+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7819fd00 con 0x7fcd78103340 2024-03-28T02:09:11.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.132+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd78108ab0 con 0x7fcd78103340 2024-03-28T02:09:11.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.134+0000 7fcd6effd640 1 -- 172.21.15.106:0/2815885271 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcd74044020 con 0x7fcd78103340 2024-03-28T02:09:11.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.135+0000 7fcd6effd640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd640a2800 0x7fcd640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.135+0000 7fcd7dcfe640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd640a2800 0x7fcd640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.135+0000 7fcd6effd640 1 -- 172.21.15.106:0/2815885271 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcd740f0e70 con 0x7fcd78103340 2024-03-28T02:09:11.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.135+0000 7fcd7dcfe640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd640a2800 0x7fcd640a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcd700095f0 tx=0x7fcd70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:11.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.138+0000 7fcd6effd640 1 -- 172.21.15.106:0/2815885271 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcd7404b020 con 0x7fcd78103340 2024-03-28T02:09:11.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.286+0000 7fcd6effd640 1 -- 172.21.15.106:0/2815885271 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fcd740b9c40 con 0x7fcd78103340 2024-03-28T02:09:11.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:11.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:11.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] up:active 2024-03-28T02:09:11.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T02:09:11.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T02:09:11.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: Cluster is now healthy 2024-03-28T02:09:11.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T02:09:11.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-28T02:09:11.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-03-28T02:09:11.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:11.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:11 smithi106 ceph-mon[31965]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 511 B/s wr, 2 op/s 2024-03-28T02:09:11.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.361+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd7802d070 con 0x7fcd78103340 2024-03-28T02:09:11.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.362+0000 7fcd6effd640 1 -- 172.21.15.106:0/2815885271 <== mon.0 v2:172.21.15.106:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11474 (secure 0 0 0) 0x7fcd740bdcd0 con 0x7fcd78103340 2024-03-28T02:09:11.356 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:11.356 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:09:11.288485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24499,"mds_2":24460,"mds_3":24446,"mds_4":24415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14625":{"gid":14625,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14637":{"gid":14637,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24446":{"gid":24446,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24415,24499,24509,24446,24460]},"id":1}]} 2024-03-28T02:09:11.356 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 28 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd640a2800 msgr2=0x7fcd640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd640a2800 0x7fcd640a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcd700095f0 tx=0x7fcd70005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 msgr2=0x7fcd7819ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 0x7fcd7819ef60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009ca0 tx=0x7fcd740346d0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 shutdown_connections 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd78103340 0x7fcd7819ef60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd640a2800 0x7fcd640a4cc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78102980 0x7fcd78197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 --2- 172.21.15.106:0/2815885271 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd780ff9b0 0x7fcd781979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 >> 172.21.15.106:0/2815885271 conn(0x7fcd780fb2e0 msgr2=0x7fcd78105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 shutdown_connections 2024-03-28T02:09:11.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.364+0000 7fcd7ff62640 1 -- 172.21.15.106:0/2815885271 wait complete. 2024-03-28T02:09:11.368 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:09:11.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 -- 172.21.15.106:0/4081995748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481005e0 msgr2=0x7fae48100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 --2- 172.21.15.106:0/4081995748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481005e0 0x7fae48100a00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fae38009900 tx=0x7fae3802bc30 comp rx=0 tx=0).stop 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 -- 172.21.15.106:0/4081995748 shutdown_connections 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 --2- 172.21.15.106:0/4081995748 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae48102190 0x7fae48108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 --2- 172.21.15.106:0/4081995748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae481017d0 0x7fae48101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 --2- 172.21.15.106:0/4081995748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481005e0 0x7fae48100a00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 -- 172.21.15.106:0/4081995748 >> 172.21.15.106:0/4081995748 conn(0x7fae480fbb50 msgr2=0x7fae480fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 -- 172.21.15.106:0/4081995748 shutdown_connections 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 -- 172.21.15.106:0/4081995748 wait complete. 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.477+0000 7fae4e80b640 1 Processor -- start 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae4e80b640 1 -- start start 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae4e80b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae481005e0 0x7fae4806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae4e80b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481017d0 0x7fae4819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae4e80b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae48102190 0x7fae4819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae4e80b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4810c250 con 0x7fae481017d0 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae4e80b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4810c0d0 con 0x7fae48102190 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae4e80b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4810c3d0 con 0x7fae481005e0 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae477fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481017d0 0x7fae4819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae47fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae481005e0 0x7fae4806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae477fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481017d0 0x7fae4819c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35840/0 (socket says 172.21.15.106:35840) 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae477fe640 1 -- 172.21.15.106:0/69182963 learned_addr learned my addr 172.21.15.106:0/69182963 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae4cda8640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae48102190 0x7fae4819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae477fe640 1 -- 172.21.15.106:0/69182963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae481005e0 msgr2=0x7fae4806a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae477fe640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae481005e0 0x7fae4806a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae477fe640 1 -- 172.21.15.106:0/69182963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae48102190 msgr2=0x7fae4819ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae477fe640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae48102190 0x7fae4819ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae477fe640 1 -- 172.21.15.106:0/69182963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4819f650 con 0x7fae481017d0 2024-03-28T02:09:11.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.478+0000 7fae477fe640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481017d0 0x7fae4819c720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00e9f0 tx=0x7fae3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:11.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.479+0000 7fae457fa640 1 -- 172.21.15.106:0/69182963 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c00cd80 con 0x7fae481017d0 2024-03-28T02:09:11.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.479+0000 7fae457fa640 1 -- 172.21.15.106:0/69182963 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae3c010910 con 0x7fae481017d0 2024-03-28T02:09:11.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.479+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4819f8e0 con 0x7fae481017d0 2024-03-28T02:09:11.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.479+0000 7fae457fa640 1 -- 172.21.15.106:0/69182963 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c018990 con 0x7fae481017d0 2024-03-28T02:09:11.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.479+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4819fe30 con 0x7fae481017d0 2024-03-28T02:09:11.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.480+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae48107900 con 0x7fae481017d0 2024-03-28T02:09:11.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.481+0000 7fae457fa640 1 -- 172.21.15.106:0/69182963 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fae3c010ac0 con 0x7fae481017d0 2024-03-28T02:09:11.476 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.481+0000 7fae457fa640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fae300a2890 0x7fae300a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.476 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.481+0000 7fae47fff640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fae300a2890 0x7fae300a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.476 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.481+0000 7fae457fa640 1 -- 172.21.15.106:0/69182963 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fae3c0cf160 con 0x7fae481017d0 2024-03-28T02:09:11.476 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.482+0000 7fae47fff640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fae300a2890 0x7fae300a4d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fae38000c00 tx=0x7fae38005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:11.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.485+0000 7fae457fa640 1 -- 172.21.15.106:0/69182963 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae3c09b810 con 0x7fae481017d0 2024-03-28T02:09:11.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:11.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:11.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] up:active 2024-03-28T02:09:11.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T02:09:11.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T02:09:11.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: Cluster is now healthy 2024-03-28T02:09:11.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T02:09:11.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-28T02:09:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-03-28T02:09:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:11.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:11 smithi178 ceph-mon[36521]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 511 B/s wr, 2 op/s 2024-03-28T02:09:11.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.663+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae4802d2c0 con 0x7fae481017d0 2024-03-28T02:09:11.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.665+0000 7fae457fa640 1 -- 172.21.15.106:0/69182963 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7fae3c09f8b0 con 0x7fae481017d0 2024-03-28T02:09:11.658 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:11.659 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:09:11.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fae300a2890 msgr2=0x7fae300a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fae300a2890 0x7fae300a4d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fae38000c00 tx=0x7fae38005fd0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481017d0 msgr2=0x7fae4819c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481017d0 0x7fae4819c720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00e9f0 tx=0x7fae3c00eeb0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 shutdown_connections 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fae300a2890 0x7fae300a4d50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae48102190 0x7fae4819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae481017d0 0x7fae4819c720 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 --2- 172.21.15.106:0/69182963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae481005e0 0x7fae4806a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 >> 172.21.15.106:0/69182963 conn(0x7fae480fbb50 msgr2=0x7fae48104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 shutdown_connections 2024-03-28T02:09:11.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.667+0000 7fae4e80b640 1 -- 172.21.15.106:0/69182963 wait complete. 2024-03-28T02:09:11.672 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T02:09:11.672 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] up:active 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: Cluster is now healthy 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:11.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:11 smithi153 ceph-mon[36228]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 511 B/s wr, 2 op/s 2024-03-28T02:09:11.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.780+0000 7fa957fb3640 1 -- 172.21.15.106:0/2784468263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa950102170 msgr2=0x7fa950108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.780+0000 7fa957fb3640 1 --2- 172.21.15.106:0/2784468263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa950102170 0x7fa950108a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa948009ca0 tx=0x7fa94802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.780+0000 7fa957fb3640 1 -- 172.21.15.106:0/2784468263 shutdown_connections 2024-03-28T02:09:11.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.780+0000 7fa957fb3640 1 --2- 172.21.15.106:0/2784468263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa950102170 0x7fa950108a00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.780+0000 7fa957fb3640 1 --2- 172.21.15.106:0/2784468263 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9501017b0 0x7fa950101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.780+0000 7fa957fb3640 1 --2- 172.21.15.106:0/2784468263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9501005c0 0x7fa9501009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 -- 172.21.15.106:0/2784468263 >> 172.21.15.106:0/2784468263 conn(0x7fa9500fbb30 msgr2=0x7fa9500fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:11.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 -- 172.21.15.106:0/2784468263 shutdown_connections 2024-03-28T02:09:11.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 -- 172.21.15.106:0/2784468263 wait complete. 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 Processor -- start 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 -- start start 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9501005c0 0x7fa9501979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9501017b0 0x7fa950197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa950102170 0x7fa95019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95010c240 con 0x7fa950102170 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95010c0c0 con 0x7fa9501005c0 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa957fb3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95010c3c0 con 0x7fa9501017b0 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa956550640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa950102170 0x7fa95019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa95554e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9501017b0 0x7fa950197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa956550640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa950102170 0x7fa95019ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35856/0 (socket says 172.21.15.106:35856) 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa955d4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9501005c0 0x7fa9501979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa95554e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9501017b0 0x7fa950197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:35040/0 (socket says 172.21.15.106:35040) 2024-03-28T02:09:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.781+0000 7fa956550640 1 -- 172.21.15.106:0/3734782350 learned_addr learned my addr 172.21.15.106:0/3734782350 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa956550640 1 -- 172.21.15.106:0/3734782350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9501017b0 msgr2=0x7fa950197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa956550640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9501017b0 0x7fa950197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa956550640 1 -- 172.21.15.106:0/3734782350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9501005c0 msgr2=0x7fa9501979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa956550640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9501005c0 0x7fa9501979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa956550640 1 -- 172.21.15.106:0/3734782350 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95019f590 con 0x7fa950102170 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa956550640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa950102170 0x7fa95019ef20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa948037040 tx=0x7fa948032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa946ffd640 1 -- 172.21.15.106:0/3734782350 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa948041020 con 0x7fa950102170 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95019f820 con 0x7fa950102170 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa946ffd640 1 -- 172.21.15.106:0/3734782350 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa948034cd0 con 0x7fa950102170 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa946ffd640 1 -- 172.21.15.106:0/3734782350 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa948039040 con 0x7fa950102170 2024-03-28T02:09:11.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.782+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95019fd60 con 0x7fa950102170 2024-03-28T02:09:11.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.783+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9501a0020 con 0x7fa950102170 2024-03-28T02:09:11.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.784+0000 7fa946ffd640 1 -- 172.21.15.106:0/3734782350 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa948044020 con 0x7fa950102170 2024-03-28T02:09:11.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.785+0000 7fa946ffd640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa93c0a2890 0x7fa93c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:11.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.785+0000 7fa955d4f640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa93c0a2890 0x7fa93c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:11.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.785+0000 7fa946ffd640 1 -- 172.21.15.106:0/3734782350 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa9480f0bb0 con 0x7fa950102170 2024-03-28T02:09:11.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.785+0000 7fa955d4f640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa93c0a2890 0x7fa93c0a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa9400095f0 tx=0x7fa940005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:11.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.789+0000 7fa946ffd640 1 -- 172.21.15.106:0/3734782350 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa94804b020 con 0x7fa950102170 2024-03-28T02:09:11.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.960+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9500fdc60 con 0x7fa93c0a2890 2024-03-28T02:09:11.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.970+0000 7fa946ffd640 1 -- 172.21.15.106:0/3734782350 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340500 (secure 0 0 0) 0x7fa9500fdc60 con 0x7fa93c0a2890 2024-03-28T02:09:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:11.969 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-03-28T02:09:11.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa93c0a2890 msgr2=0x7fa93c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa93c0a2890 0x7fa93c0a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa9400095f0 tx=0x7fa940005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:11.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa950102170 msgr2=0x7fa95019ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:11.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa950102170 0x7fa95019ef20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa948037040 tx=0x7fa948032f40 comp rx=0 tx=0).stop 2024-03-28T02:09:11.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 shutdown_connections 2024-03-28T02:09:11.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa950102170 0x7fa95019ef20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9501017b0 0x7fa950197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa93c0a2890 0x7fa93c0a4d50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 --2- 172.21.15.106:0/3734782350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9501005c0 0x7fa9501979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:11.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.973+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 >> 172.21.15.106:0/3734782350 conn(0x7fa9500fbb30 msgr2=0x7fa9500fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:11.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.974+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 shutdown_connections 2024-03-28T02:09:11.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:11.974+0000 7fa957fb3640 1 -- 172.21.15.106:0/3734782350 wait complete. 2024-03-28T02:09:11.978 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-28T02:09:10.355899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595098,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":142,"num_read_kb":127,"num_write":249,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377380,"kb_used_data":11548,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564444,"statfs":{"total":1151940427776,"available":1151553990656,"internally_reserved":0,"allocated":11825152,"data_stored":7513592,"data_compressed":284361,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2320,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":23,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003188"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873745+0000","last_change":"2024-03-28T02:08:49.268140+0000","last_active":"2024-03-28T02:08:55.873745+0000","last_peered":"2024-03-28T02:08:55.873745+0000","last_clean":"2024-03-28T02:08:55.873745+0000","last_became_active":"2024-03-28T02:08:49.267695+0000","last_became_peered":"2024-03-28T02:08:49.267695+0000","last_unstale":"2024-03-28T02:08:55.873745+0000","last_undegraded":"2024-03-28T02:08:55.873745+0000","last_fullsized":"2024-03-28T02:08:55.873745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:27:09.768237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734162+0000","last_change":"2024-03-28T02:08:49.259914+0000","last_active":"2024-03-28T02:08:55.734162+0000","last_peered":"2024-03-28T02:08:55.734162+0000","last_clean":"2024-03-28T02:08:55.734162+0000","last_became_active":"2024-03-28T02:08:49.259579+0000","last_became_peered":"2024-03-28T02:08:49.259579+0000","last_unstale":"2024-03-28T02:08:55.734162+0000","last_undegraded":"2024-03-28T02:08:55.734162+0000","last_fullsized":"2024-03-28T02:08:55.734162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:04:02.664429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734021+0000","last_change":"2024-03-28T02:08:49.268234+0000","last_active":"2024-03-28T02:08:55.734021+0000","last_peered":"2024-03-28T02:08:55.734021+0000","last_clean":"2024-03-28T02:08:55.734021+0000","last_became_active":"2024-03-28T02:08:49.267813+0000","last_became_peered":"2024-03-28T02:08:49.267813+0000","last_unstale":"2024-03-28T02:08:55.734021+0000","last_undegraded":"2024-03-28T02:08:55.734021+0000","last_fullsized":"2024-03-28T02:08:55.734021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:46:21.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873447+0000","last_change":"2024-03-28T02:08:49.268397+0000","last_active":"2024-03-28T02:08:55.873447+0000","last_peered":"2024-03-28T02:08:55.873447+0000","last_clean":"2024-03-28T02:08:55.873447+0000","last_became_active":"2024-03-28T02:08:49.268059+0000","last_became_peered":"2024-03-28T02:08:49.268059+0000","last_unstale":"2024-03-28T02:08:55.873447+0000","last_undegraded":"2024-03-28T02:08:55.873447+0000","last_fullsized":"2024-03-28T02:08:55.873447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:45:51.951306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873348+0000","last_change":"2024-03-28T02:08:49.257827+0000","last_active":"2024-03-28T02:08:55.873348+0000","last_peered":"2024-03-28T02:08:55.873348+0000","last_clean":"2024-03-28T02:08:55.873348+0000","last_became_active":"2024-03-28T02:08:49.256707+0000","last_became_peered":"2024-03-28T02:08:49.256707+0000","last_unstale":"2024-03-28T02:08:55.873348+0000","last_undegraded":"2024-03-28T02:08:55.873348+0000","last_fullsized":"2024-03-28T02:08:55.873348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:23:13.617900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482650+0000","last_change":"2024-03-28T02:08:49.262545+0000","last_active":"2024-03-28T02:08:54.482650+0000","last_peered":"2024-03-28T02:08:54.482650+0000","last_clean":"2024-03-28T02:08:54.482650+0000","last_became_active":"2024-03-28T02:08:49.262218+0000","last_became_peered":"2024-03-28T02:08:49.262218+0000","last_unstale":"2024-03-28T02:08:54.482650+0000","last_undegraded":"2024-03-28T02:08:54.482650+0000","last_fullsized":"2024-03-28T02:08:54.482650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:55:20.418608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873302+0000","last_change":"2024-03-28T02:08:49.260072+0000","last_active":"2024-03-28T02:08:55.873302+0000","last_peered":"2024-03-28T02:08:55.873302+0000","last_clean":"2024-03-28T02:08:55.873302+0000","last_became_active":"2024-03-28T02:08:49.259724+0000","last_became_peered":"2024-03-28T02:08:49.259724+0000","last_unstale":"2024-03-28T02:08:55.873302+0000","last_undegraded":"2024-03-28T02:08:55.873302+0000","last_fullsized":"2024-03-28T02:08:55.873302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:05.445467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872432+0000","last_change":"2024-03-28T02:08:49.262547+0000","last_active":"2024-03-28T02:08:55.872432+0000","last_peered":"2024-03-28T02:08:55.872432+0000","last_clean":"2024-03-28T02:08:55.872432+0000","last_became_active":"2024-03-28T02:08:49.261836+0000","last_became_peered":"2024-03-28T02:08:49.261836+0000","last_unstale":"2024-03-28T02:08:55.872432+0000","last_undegraded":"2024-03-28T02:08:55.872432+0000","last_fullsized":"2024-03-28T02:08:55.872432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:55:22.201524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760658+0000","last_change":"2024-03-28T02:08:49.262725+0000","last_active":"2024-03-28T02:08:54.760658+0000","last_peered":"2024-03-28T02:08:54.760658+0000","last_clean":"2024-03-28T02:08:54.760658+0000","last_became_active":"2024-03-28T02:08:49.262319+0000","last_became_peered":"2024-03-28T02:08:49.262319+0000","last_unstale":"2024-03-28T02:08:54.760658+0000","last_undegraded":"2024-03-28T02:08:54.760658+0000","last_fullsized":"2024-03-28T02:08:54.760658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:56:24.904009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483037+0000","last_change":"2024-03-28T02:08:49.262194+0000","last_active":"2024-03-28T02:08:54.483037+0000","last_peered":"2024-03-28T02:08:54.483037+0000","last_clean":"2024-03-28T02:08:54.483037+0000","last_became_active":"2024-03-28T02:08:49.261887+0000","last_became_peered":"2024-03-28T02:08:49.261887+0000","last_unstale":"2024-03-28T02:08:54.483037+0000","last_undegraded":"2024-03-28T02:08:54.483037+0000","last_fullsized":"2024-03-28T02:08:54.483037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:52:47.049430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873632+0000","last_change":"2024-03-28T02:08:49.269222+0000","last_active":"2024-03-28T02:08:55.873632+0000","last_peered":"2024-03-28T02:08:55.873632+0000","last_clean":"2024-03-28T02:08:55.873632+0000","last_became_active":"2024-03-28T02:08:49.268647+0000","last_became_peered":"2024-03-28T02:08:49.268647+0000","last_unstale":"2024-03-28T02:08:55.873632+0000","last_undegraded":"2024-03-28T02:08:55.873632+0000","last_fullsized":"2024-03-28T02:08:55.873632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:10:01.108158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873345+0000","last_change":"2024-03-28T02:08:49.269868+0000","last_active":"2024-03-28T02:08:55.873345+0000","last_peered":"2024-03-28T02:08:55.873345+0000","last_clean":"2024-03-28T02:08:55.873345+0000","last_became_active":"2024-03-28T02:08:49.269035+0000","last_became_peered":"2024-03-28T02:08:49.269035+0000","last_unstale":"2024-03-28T02:08:55.873345+0000","last_undegraded":"2024-03-28T02:08:55.873345+0000","last_fullsized":"2024-03-28T02:08:55.873345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:31.153970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293465+0000","last_change":"2024-03-28T02:08:49.254162+0000","last_active":"2024-03-28T02:08:55.293465+0000","last_peered":"2024-03-28T02:08:55.293465+0000","last_clean":"2024-03-28T02:08:55.293465+0000","last_became_active":"2024-03-28T02:08:49.253756+0000","last_became_peered":"2024-03-28T02:08:49.253756+0000","last_unstale":"2024-03-28T02:08:55.293465+0000","last_undegraded":"2024-03-28T02:08:55.293465+0000","last_fullsized":"2024-03-28T02:08:55.293465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:22:20.791809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.733751+0000","last_change":"2024-03-28T02:08:49.257940+0000","last_active":"2024-03-28T02:08:55.733751+0000","last_peered":"2024-03-28T02:08:55.733751+0000","last_clean":"2024-03-28T02:08:55.733751+0000","last_became_active":"2024-03-28T02:08:49.257039+0000","last_became_peered":"2024-03-28T02:08:49.257039+0000","last_unstale":"2024-03-28T02:08:55.733751+0000","last_undegraded":"2024-03-28T02:08:55.733751+0000","last_fullsized":"2024-03-28T02:08:55.733751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:36:10.701760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.733744+0000","last_change":"2024-03-28T02:08:49.267502+0000","last_active":"2024-03-28T02:08:55.733744+0000","last_peered":"2024-03-28T02:08:55.733744+0000","last_clean":"2024-03-28T02:08:55.733744+0000","last_became_active":"2024-03-28T02:08:49.265447+0000","last_became_peered":"2024-03-28T02:08:49.265447+0000","last_unstale":"2024-03-28T02:08:55.733744+0000","last_undegraded":"2024-03-28T02:08:55.733744+0000","last_fullsized":"2024-03-28T02:08:55.733744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:47:44.891440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760339+0000","last_change":"2024-03-28T02:08:49.267702+0000","last_active":"2024-03-28T02:08:54.760339+0000","last_peered":"2024-03-28T02:08:54.760339+0000","last_clean":"2024-03-28T02:08:54.760339+0000","last_became_active":"2024-03-28T02:08:49.267381+0000","last_became_peered":"2024-03-28T02:08:49.267381+0000","last_unstale":"2024-03-28T02:08:54.760339+0000","last_undegraded":"2024-03-28T02:08:54.760339+0000","last_fullsized":"2024-03-28T02:08:54.760339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:55:59.904141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873564+0000","last_change":"2024-03-28T02:08:49.268240+0000","last_active":"2024-03-28T02:08:55.873564+0000","last_peered":"2024-03-28T02:08:55.873564+0000","last_clean":"2024-03-28T02:08:55.873564+0000","last_became_active":"2024-03-28T02:08:49.267884+0000","last_became_peered":"2024-03-28T02:08:49.267884+0000","last_unstale":"2024-03-28T02:08:55.873564+0000","last_undegraded":"2024-03-28T02:08:55.873564+0000","last_fullsized":"2024-03-28T02:08:55.873564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:34:23.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482698+0000","last_change":"2024-03-28T02:08:49.262585+0000","last_active":"2024-03-28T02:08:54.482698+0000","last_peered":"2024-03-28T02:08:54.482698+0000","last_clean":"2024-03-28T02:08:54.482698+0000","last_became_active":"2024-03-28T02:08:49.262217+0000","last_became_peered":"2024-03-28T02:08:49.262217+0000","last_unstale":"2024-03-28T02:08:54.482698+0000","last_undegraded":"2024-03-28T02:08:54.482698+0000","last_fullsized":"2024-03-28T02:08:54.482698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:10:05.714733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483506+0000","last_change":"2024-03-28T02:08:49.261550+0000","last_active":"2024-03-28T02:08:54.483506+0000","last_peered":"2024-03-28T02:08:54.483506+0000","last_clean":"2024-03-28T02:08:54.483506+0000","last_became_active":"2024-03-28T02:08:49.261196+0000","last_became_peered":"2024-03-28T02:08:49.261196+0000","last_unstale":"2024-03-28T02:08:54.483506+0000","last_undegraded":"2024-03-28T02:08:54.483506+0000","last_fullsized":"2024-03-28T02:08:54.483506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:52:52.972048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483056+0000","last_change":"2024-03-28T02:08:53.491877+0000","last_active":"2024-03-28T02:08:54.483056+0000","last_peered":"2024-03-28T02:08:54.483056+0000","last_clean":"2024-03-28T02:08:54.483056+0000","last_became_active":"2024-03-28T02:08:53.491584+0000","last_became_peered":"2024-03-28T02:08:53.491584+0000","last_unstale":"2024-03-28T02:08:54.483056+0000","last_undegraded":"2024-03-28T02:08:54.483056+0000","last_fullsized":"2024-03-28T02:08:54.483056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:47:41.778839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:08.262424+0000","last_change":"2024-03-28T02:08:48.448277+0000","last_active":"2024-03-28T02:09:08.262424+0000","last_peered":"2024-03-28T02:09:08.262424+0000","last_clean":"2024-03-28T02:09:08.262424+0000","last_became_active":"2024-03-28T02:08:48.447910+0000","last_became_peered":"2024-03-28T02:08:48.447910+0000","last_unstale":"2024-03-28T02:09:08.262424+0000","last_undegraded":"2024-03-28T02:09:08.262424+0000","last_fullsized":"2024-03-28T02:09:08.262424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:13:34.241952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873068+0000","last_change":"2024-03-28T02:08:49.258840+0000","last_active":"2024-03-28T02:08:55.873068+0000","last_peered":"2024-03-28T02:08:55.873068+0000","last_clean":"2024-03-28T02:08:55.873068+0000","last_became_active":"2024-03-28T02:08:49.258259+0000","last_became_peered":"2024-03-28T02:08:49.258259+0000","last_unstale":"2024-03-28T02:08:55.873068+0000","last_undegraded":"2024-03-28T02:08:55.873068+0000","last_fullsized":"2024-03-28T02:08:55.873068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:25:31.170413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872856+0000","last_change":"2024-03-28T02:08:53.496476+0000","last_active":"2024-03-28T02:08:55.872856+0000","last_peered":"2024-03-28T02:08:55.872856+0000","last_clean":"2024-03-28T02:08:55.872856+0000","last_became_active":"2024-03-28T02:08:53.496034+0000","last_became_peered":"2024-03-28T02:08:53.496034+0000","last_unstale":"2024-03-28T02:08:55.872856+0000","last_undegraded":"2024-03-28T02:08:55.872856+0000","last_fullsized":"2024-03-28T02:08:55.872856+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:49:00.550877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873186+0000","last_change":"2024-03-28T02:08:48.457960+0000","last_active":"2024-03-28T02:08:55.873186+0000","last_peered":"2024-03-28T02:08:55.873186+0000","last_clean":"2024-03-28T02:08:55.873186+0000","last_became_active":"2024-03-28T02:08:48.455528+0000","last_became_peered":"2024-03-28T02:08:48.455528+0000","last_unstale":"2024-03-28T02:08:55.873186+0000","last_undegraded":"2024-03-28T02:08:55.873186+0000","last_fullsized":"2024-03-28T02:08:55.873186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:40:37.830166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734036+0000","last_change":"2024-03-28T02:08:49.268390+0000","last_active":"2024-03-28T02:08:55.734036+0000","last_peered":"2024-03-28T02:08:55.734036+0000","last_clean":"2024-03-28T02:08:55.734036+0000","last_became_active":"2024-03-28T02:08:49.268025+0000","last_became_peered":"2024-03-28T02:08:49.268025+0000","last_unstale":"2024-03-28T02:08:55.734036+0000","last_undegraded":"2024-03-28T02:08:55.734036+0000","last_fullsized":"2024-03-28T02:08:55.734036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:40:07.752553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873355+0000","last_change":"2024-03-28T02:08:53.492654+0000","last_active":"2024-03-28T02:08:55.873355+0000","last_peered":"2024-03-28T02:08:55.873355+0000","last_clean":"2024-03-28T02:08:55.873355+0000","last_became_active":"2024-03-28T02:08:53.491994+0000","last_became_peered":"2024-03-28T02:08:53.491994+0000","last_unstale":"2024-03-28T02:08:55.873355+0000","last_undegraded":"2024-03-28T02:08:55.873355+0000","last_fullsized":"2024-03-28T02:08:55.873355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:30:55.365074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873730+0000","last_change":"2024-03-28T02:08:48.459667+0000","last_active":"2024-03-28T02:08:55.873730+0000","last_peered":"2024-03-28T02:08:55.873730+0000","last_clean":"2024-03-28T02:08:55.873730+0000","last_became_active":"2024-03-28T02:08:48.459310+0000","last_became_peered":"2024-03-28T02:08:48.459310+0000","last_unstale":"2024-03-28T02:08:55.873730+0000","last_undegraded":"2024-03-28T02:08:55.873730+0000","last_fullsized":"2024-03-28T02:08:55.873730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:51:07.351151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760704+0000","last_change":"2024-03-28T02:08:49.254576+0000","last_active":"2024-03-28T02:08:54.760704+0000","last_peered":"2024-03-28T02:08:54.760704+0000","last_clean":"2024-03-28T02:08:54.760704+0000","last_became_active":"2024-03-28T02:08:49.254213+0000","last_became_peered":"2024-03-28T02:08:49.254213+0000","last_unstale":"2024-03-28T02:08:54.760704+0000","last_undegraded":"2024-03-28T02:08:54.760704+0000","last_fullsized":"2024-03-28T02:08:54.760704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.630801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873534+0000","last_change":"2024-03-28T02:08:53.493624+0000","last_active":"2024-03-28T02:08:55.873534+0000","last_peered":"2024-03-28T02:08:55.873534+0000","last_clean":"2024-03-28T02:08:55.873534+0000","last_became_active":"2024-03-28T02:08:53.493195+0000","last_became_peered":"2024-03-28T02:08:53.493195+0000","last_unstale":"2024-03-28T02:08:55.873534+0000","last_undegraded":"2024-03-28T02:08:55.873534+0000","last_fullsized":"2024-03-28T02:08:55.873534+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:38:25.497464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873375+0000","last_change":"2024-03-28T02:08:48.454008+0000","last_active":"2024-03-28T02:08:55.873375+0000","last_peered":"2024-03-28T02:08:55.873375+0000","last_clean":"2024-03-28T02:08:55.873375+0000","last_became_active":"2024-03-28T02:08:48.450674+0000","last_became_peered":"2024-03-28T02:08:48.450674+0000","last_unstale":"2024-03-28T02:08:55.873375+0000","last_undegraded":"2024-03-28T02:08:55.873375+0000","last_fullsized":"2024-03-28T02:08:55.873375+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:08:12.902348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611109+0000","last_change":"2024-03-28T02:08:49.266552+0000","last_active":"2024-03-28T02:08:55.611109+0000","last_peered":"2024-03-28T02:08:55.611109+0000","last_clean":"2024-03-28T02:08:55.611109+0000","last_became_active":"2024-03-28T02:08:49.265161+0000","last_became_peered":"2024-03-28T02:08:49.265161+0000","last_unstale":"2024-03-28T02:08:55.611109+0000","last_undegraded":"2024-03-28T02:08:55.611109+0000","last_fullsized":"2024-03-28T02:08:55.611109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:22.128908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483449+0000","last_change":"2024-03-28T02:08:53.487871+0000","last_active":"2024-03-28T02:08:54.483449+0000","last_peered":"2024-03-28T02:08:54.483449+0000","last_clean":"2024-03-28T02:08:54.483449+0000","last_became_active":"2024-03-28T02:08:53.487344+0000","last_became_peered":"2024-03-28T02:08:53.487344+0000","last_unstale":"2024-03-28T02:08:54.483449+0000","last_undegraded":"2024-03-28T02:08:54.483449+0000","last_fullsized":"2024-03-28T02:08:54.483449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:34:12.555549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610867+0000","last_change":"2024-03-28T02:08:48.450507+0000","last_active":"2024-03-28T02:08:55.610867+0000","last_peered":"2024-03-28T02:08:55.610867+0000","last_clean":"2024-03-28T02:08:55.610867+0000","last_became_active":"2024-03-28T02:08:48.450065+0000","last_became_peered":"2024-03-28T02:08:48.450065+0000","last_unstale":"2024-03-28T02:08:55.610867+0000","last_undegraded":"2024-03-28T02:08:55.610867+0000","last_fullsized":"2024-03-28T02:08:55.610867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:00:45.393364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873190+0000","last_change":"2024-03-28T02:08:49.256313+0000","last_active":"2024-03-28T02:08:55.873190+0000","last_peered":"2024-03-28T02:08:55.873190+0000","last_clean":"2024-03-28T02:08:55.873190+0000","last_became_active":"2024-03-28T02:08:49.255957+0000","last_became_peered":"2024-03-28T02:08:49.255957+0000","last_unstale":"2024-03-28T02:08:55.873190+0000","last_undegraded":"2024-03-28T02:08:55.873190+0000","last_fullsized":"2024-03-28T02:08:55.873190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:39:27.402983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483351+0000","last_change":"2024-03-28T02:08:53.498554+0000","last_active":"2024-03-28T02:08:54.483351+0000","last_peered":"2024-03-28T02:08:54.483351+0000","last_clean":"2024-03-28T02:08:54.483351+0000","last_became_active":"2024-03-28T02:08:53.498152+0000","last_became_peered":"2024-03-28T02:08:53.498152+0000","last_unstale":"2024-03-28T02:08:54.483351+0000","last_undegraded":"2024-03-28T02:08:54.483351+0000","last_fullsized":"2024-03-28T02:08:54.483351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:41:38.234758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:08.261134+0000","last_change":"2024-03-28T02:08:48.450494+0000","last_active":"2024-03-28T02:09:08.261134+0000","last_peered":"2024-03-28T02:09:08.261134+0000","last_clean":"2024-03-28T02:09:08.261134+0000","last_became_active":"2024-03-28T02:08:48.450158+0000","last_became_peered":"2024-03-28T02:08:48.450158+0000","last_unstale":"2024-03-28T02:09:08.261134+0000","last_undegraded":"2024-03-28T02:09:08.261134+0000","last_fullsized":"2024-03-28T02:09:08.261134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:47:20.021116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611030+0000","last_change":"2024-03-28T02:08:49.265941+0000","last_active":"2024-03-28T02:08:55.611030+0000","last_peered":"2024-03-28T02:08:55.611030+0000","last_clean":"2024-03-28T02:08:55.611030+0000","last_became_active":"2024-03-28T02:08:49.262662+0000","last_became_peered":"2024-03-28T02:08:49.262662+0000","last_unstale":"2024-03-28T02:08:55.611030+0000","last_undegraded":"2024-03-28T02:08:55.611030+0000","last_fullsized":"2024-03-28T02:08:55.611030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:07:23.438391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873535+0000","last_change":"2024-03-28T02:08:53.490046+0000","last_active":"2024-03-28T02:08:55.873535+0000","last_peered":"2024-03-28T02:08:55.873535+0000","last_clean":"2024-03-28T02:08:55.873535+0000","last_became_active":"2024-03-28T02:08:53.489463+0000","last_became_peered":"2024-03-28T02:08:53.489463+0000","last_unstale":"2024-03-28T02:08:55.873535+0000","last_undegraded":"2024-03-28T02:08:55.873535+0000","last_fullsized":"2024-03-28T02:08:55.873535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:25:03.485568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293277+0000","last_change":"2024-03-28T02:08:48.450548+0000","last_active":"2024-03-28T02:08:55.293277+0000","last_peered":"2024-03-28T02:08:55.293277+0000","last_clean":"2024-03-28T02:08:55.293277+0000","last_became_active":"2024-03-28T02:08:48.450068+0000","last_became_peered":"2024-03-28T02:08:48.450068+0000","last_unstale":"2024-03-28T02:08:55.293277+0000","last_undegraded":"2024-03-28T02:08:55.293277+0000","last_fullsized":"2024-03-28T02:08:55.293277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:24:05.231774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872623+0000","last_change":"2024-03-28T02:08:49.262785+0000","last_active":"2024-03-28T02:08:55.872623+0000","last_peered":"2024-03-28T02:08:55.872623+0000","last_clean":"2024-03-28T02:08:55.872623+0000","last_became_active":"2024-03-28T02:08:49.262253+0000","last_became_peered":"2024-03-28T02:08:49.262253+0000","last_unstale":"2024-03-28T02:08:55.872623+0000","last_undegraded":"2024-03-28T02:08:55.872623+0000","last_fullsized":"2024-03-28T02:08:55.872623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:07:37.064693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734109+0000","last_change":"2024-03-28T02:08:53.483738+0000","last_active":"2024-03-28T02:08:55.734109+0000","last_peered":"2024-03-28T02:08:55.734109+0000","last_clean":"2024-03-28T02:08:55.734109+0000","last_became_active":"2024-03-28T02:08:53.483214+0000","last_became_peered":"2024-03-28T02:08:53.483214+0000","last_unstale":"2024-03-28T02:08:55.734109+0000","last_undegraded":"2024-03-28T02:08:55.734109+0000","last_fullsized":"2024-03-28T02:08:55.734109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:06:32.789839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610462+0000","last_change":"2024-03-28T02:08:48.448858+0000","last_active":"2024-03-28T02:08:55.610462+0000","last_peered":"2024-03-28T02:08:55.610462+0000","last_clean":"2024-03-28T02:08:55.610462+0000","last_became_active":"2024-03-28T02:08:48.448435+0000","last_became_peered":"2024-03-28T02:08:48.448435+0000","last_unstale":"2024-03-28T02:08:55.610462+0000","last_undegraded":"2024-03-28T02:08:55.610462+0000","last_fullsized":"2024-03-28T02:08:55.610462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:34:13.035336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482224+0000","last_change":"2024-03-28T02:08:49.261354+0000","last_active":"2024-03-28T02:08:54.482224+0000","last_peered":"2024-03-28T02:08:54.482224+0000","last_clean":"2024-03-28T02:08:54.482224+0000","last_became_active":"2024-03-28T02:08:49.261038+0000","last_became_peered":"2024-03-28T02:08:49.261038+0000","last_unstale":"2024-03-28T02:08:54.482224+0000","last_undegraded":"2024-03-28T02:08:54.482224+0000","last_fullsized":"2024-03-28T02:08:54.482224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:03:15.615484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872750+0000","last_change":"2024-03-28T02:08:53.488573+0000","last_active":"2024-03-28T02:08:55.872750+0000","last_peered":"2024-03-28T02:08:55.872750+0000","last_clean":"2024-03-28T02:08:55.872750+0000","last_became_active":"2024-03-28T02:08:53.488198+0000","last_became_peered":"2024-03-28T02:08:53.488198+0000","last_unstale":"2024-03-28T02:08:55.872750+0000","last_undegraded":"2024-03-28T02:08:55.872750+0000","last_fullsized":"2024-03-28T02:08:55.872750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:43:29.158223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:07.533049+0000","last_change":"2024-03-28T02:08:48.445941+0000","last_active":"2024-03-28T02:09:07.533049+0000","last_peered":"2024-03-28T02:09:07.533049+0000","last_clean":"2024-03-28T02:09:07.533049+0000","last_became_active":"2024-03-28T02:08:48.445599+0000","last_became_peered":"2024-03-28T02:08:48.445599+0000","last_unstale":"2024-03-28T02:09:07.533049+0000","last_undegraded":"2024-03-28T02:09:07.533049+0000","last_fullsized":"2024-03-28T02:09:07.533049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:28:53.524708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760857+0000","last_change":"2024-03-28T02:08:49.258592+0000","last_active":"2024-03-28T02:08:54.760857+0000","last_peered":"2024-03-28T02:08:54.760857+0000","last_clean":"2024-03-28T02:08:54.760857+0000","last_became_active":"2024-03-28T02:08:49.256527+0000","last_became_peered":"2024-03-28T02:08:49.256527+0000","last_unstale":"2024-03-28T02:08:54.760857+0000","last_undegraded":"2024-03-28T02:08:54.760857+0000","last_fullsized":"2024-03-28T02:08:54.760857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:54:41.645210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482255+0000","last_change":"2024-03-28T02:08:53.494677+0000","last_active":"2024-03-28T02:08:54.482255+0000","last_peered":"2024-03-28T02:08:54.482255+0000","last_clean":"2024-03-28T02:08:54.482255+0000","last_became_active":"2024-03-28T02:08:53.493076+0000","last_became_peered":"2024-03-28T02:08:53.493076+0000","last_unstale":"2024-03-28T02:08:54.482255+0000","last_undegraded":"2024-03-28T02:08:54.482255+0000","last_fullsized":"2024-03-28T02:08:54.482255+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:10:44.817473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873285+0000","last_change":"2024-03-28T02:08:48.454321+0000","last_active":"2024-03-28T02:08:55.873285+0000","last_peered":"2024-03-28T02:08:55.873285+0000","last_clean":"2024-03-28T02:08:55.873285+0000","last_became_active":"2024-03-28T02:08:48.453907+0000","last_became_peered":"2024-03-28T02:08:48.453907+0000","last_unstale":"2024-03-28T02:08:55.873285+0000","last_undegraded":"2024-03-28T02:08:55.873285+0000","last_fullsized":"2024-03-28T02:08:55.873285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:45:18.347073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734156+0000","last_change":"2024-03-28T02:08:49.262196+0000","last_active":"2024-03-28T02:08:55.734156+0000","last_peered":"2024-03-28T02:08:55.734156+0000","last_clean":"2024-03-28T02:08:55.734156+0000","last_became_active":"2024-03-28T02:08:49.261190+0000","last_became_peered":"2024-03-28T02:08:49.261190+0000","last_unstale":"2024-03-28T02:08:55.734156+0000","last_undegraded":"2024-03-28T02:08:55.734156+0000","last_fullsized":"2024-03-28T02:08:55.734156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:37:15.939535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873615+0000","last_change":"2024-03-28T02:08:53.492207+0000","last_active":"2024-03-28T02:08:55.873615+0000","last_peered":"2024-03-28T02:08:55.873615+0000","last_clean":"2024-03-28T02:08:55.873615+0000","last_became_active":"2024-03-28T02:08:53.491831+0000","last_became_peered":"2024-03-28T02:08:53.491831+0000","last_unstale":"2024-03-28T02:08:55.873615+0000","last_undegraded":"2024-03-28T02:08:55.873615+0000","last_fullsized":"2024-03-28T02:08:55.873615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:00.163397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610720+0000","last_change":"2024-03-28T02:08:48.450908+0000","last_active":"2024-03-28T02:08:55.610720+0000","last_peered":"2024-03-28T02:08:55.610720+0000","last_clean":"2024-03-28T02:08:55.610720+0000","last_became_active":"2024-03-28T02:08:48.450382+0000","last_became_peered":"2024-03-28T02:08:48.450382+0000","last_unstale":"2024-03-28T02:08:55.610720+0000","last_undegraded":"2024-03-28T02:08:55.610720+0000","last_fullsized":"2024-03-28T02:08:55.610720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:26:25.851416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760801+0000","last_change":"2024-03-28T02:08:49.268904+0000","last_active":"2024-03-28T02:08:54.760801+0000","last_peered":"2024-03-28T02:08:54.760801+0000","last_clean":"2024-03-28T02:08:54.760801+0000","last_became_active":"2024-03-28T02:08:49.268595+0000","last_became_peered":"2024-03-28T02:08:49.268595+0000","last_unstale":"2024-03-28T02:08:54.760801+0000","last_undegraded":"2024-03-28T02:08:54.760801+0000","last_fullsized":"2024-03-28T02:08:54.760801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:12:25.903861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482278+0000","last_change":"2024-03-28T02:08:53.489115+0000","last_active":"2024-03-28T02:08:54.482278+0000","last_peered":"2024-03-28T02:08:54.482278+0000","last_clean":"2024-03-28T02:08:54.482278+0000","last_became_active":"2024-03-28T02:08:53.488716+0000","last_became_peered":"2024-03-28T02:08:53.488716+0000","last_unstale":"2024-03-28T02:08:54.482278+0000","last_undegraded":"2024-03-28T02:08:54.482278+0000","last_fullsized":"2024-03-28T02:08:54.482278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:00:42.920205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:08.250333+0000","last_change":"2024-03-28T02:08:48.453921+0000","last_active":"2024-03-28T02:09:08.250333+0000","last_peered":"2024-03-28T02:09:08.250333+0000","last_clean":"2024-03-28T02:09:08.250333+0000","last_became_active":"2024-03-28T02:08:48.453301+0000","last_became_peered":"2024-03-28T02:08:48.453301+0000","last_unstale":"2024-03-28T02:09:08.250333+0000","last_undegraded":"2024-03-28T02:09:08.250333+0000","last_fullsized":"2024-03-28T02:09:08.250333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:57:04.543444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873383+0000","last_change":"2024-03-28T02:08:49.259942+0000","last_active":"2024-03-28T02:08:55.873383+0000","last_peered":"2024-03-28T02:08:55.873383+0000","last_clean":"2024-03-28T02:08:55.873383+0000","last_became_active":"2024-03-28T02:08:49.259602+0000","last_became_peered":"2024-03-28T02:08:49.259602+0000","last_unstale":"2024-03-28T02:08:55.873383+0000","last_undegraded":"2024-03-28T02:08:55.873383+0000","last_fullsized":"2024-03-28T02:08:55.873383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:38:05.897720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611464+0000","last_change":"2024-03-28T02:08:53.495675+0000","last_active":"2024-03-28T02:08:55.611464+0000","last_peered":"2024-03-28T02:08:55.611464+0000","last_clean":"2024-03-28T02:08:55.611464+0000","last_became_active":"2024-03-28T02:08:53.495233+0000","last_became_peered":"2024-03-28T02:08:53.495233+0000","last_unstale":"2024-03-28T02:08:55.611464+0000","last_undegraded":"2024-03-28T02:08:55.611464+0000","last_fullsized":"2024-03-28T02:08:55.611464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:43.805151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483024+0000","last_change":"2024-03-28T02:08:48.452335+0000","last_active":"2024-03-28T02:08:54.483024+0000","last_peered":"2024-03-28T02:08:54.483024+0000","last_clean":"2024-03-28T02:08:54.483024+0000","last_became_active":"2024-03-28T02:08:48.449966+0000","last_became_peered":"2024-03-28T02:08:48.449966+0000","last_unstale":"2024-03-28T02:08:54.483024+0000","last_undegraded":"2024-03-28T02:08:54.483024+0000","last_fullsized":"2024-03-28T02:08:54.483024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:20:09.757746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293426+0000","last_change":"2024-03-28T02:08:49.259179+0000","last_active":"2024-03-28T02:08:55.293426+0000","last_peered":"2024-03-28T02:08:55.293426+0000","last_clean":"2024-03-28T02:08:55.293426+0000","last_became_active":"2024-03-28T02:08:49.258538+0000","last_became_peered":"2024-03-28T02:08:49.258538+0000","last_unstale":"2024-03-28T02:08:55.293426+0000","last_undegraded":"2024-03-28T02:08:55.293426+0000","last_fullsized":"2024-03-28T02:08:55.293426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:12:20.008528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873479+0000","last_change":"2024-03-28T02:08:53.499523+0000","last_active":"2024-03-28T02:08:55.873479+0000","last_peered":"2024-03-28T02:08:55.873479+0000","last_clean":"2024-03-28T02:08:55.873479+0000","last_became_active":"2024-03-28T02:08:53.499112+0000","last_became_peered":"2024-03-28T02:08:53.499112+0000","last_unstale":"2024-03-28T02:08:55.873479+0000","last_undegraded":"2024-03-28T02:08:55.873479+0000","last_fullsized":"2024-03-28T02:08:55.873479+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:31:12.008640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610781+0000","last_change":"2024-03-28T02:08:48.452952+0000","last_active":"2024-03-28T02:08:55.610781+0000","last_peered":"2024-03-28T02:08:55.610781+0000","last_clean":"2024-03-28T02:08:55.610781+0000","last_became_active":"2024-03-28T02:08:48.452586+0000","last_became_peered":"2024-03-28T02:08:48.452586+0000","last_unstale":"2024-03-28T02:08:55.610781+0000","last_undegraded":"2024-03-28T02:08:55.610781+0000","last_fullsized":"2024-03-28T02:08:55.610781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:44.749902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873723+0000","last_change":"2024-03-28T02:08:49.256746+0000","last_active":"2024-03-28T02:08:55.873723+0000","last_peered":"2024-03-28T02:08:55.873723+0000","last_clean":"2024-03-28T02:08:55.873723+0000","last_became_active":"2024-03-28T02:08:49.256398+0000","last_became_peered":"2024-03-28T02:08:49.256398+0000","last_unstale":"2024-03-28T02:08:55.873723+0000","last_undegraded":"2024-03-28T02:08:55.873723+0000","last_fullsized":"2024-03-28T02:08:55.873723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:10:52.021218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482830+0000","last_change":"2024-03-28T02:08:53.494257+0000","last_active":"2024-03-28T02:08:54.482830+0000","last_peered":"2024-03-28T02:08:54.482830+0000","last_clean":"2024-03-28T02:08:54.482830+0000","last_became_active":"2024-03-28T02:08:53.493889+0000","last_became_peered":"2024-03-28T02:08:53.493889+0000","last_unstale":"2024-03-28T02:08:54.482830+0000","last_undegraded":"2024-03-28T02:08:54.482830+0000","last_fullsized":"2024-03-28T02:08:54.482830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:49.292686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872868+0000","last_change":"2024-03-28T02:08:48.447361+0000","last_active":"2024-03-28T02:08:55.872868+0000","last_peered":"2024-03-28T02:08:55.872868+0000","last_clean":"2024-03-28T02:08:55.872868+0000","last_became_active":"2024-03-28T02:08:48.446943+0000","last_became_peered":"2024-03-28T02:08:48.446943+0000","last_unstale":"2024-03-28T02:08:55.872868+0000","last_undegraded":"2024-03-28T02:08:55.872868+0000","last_fullsized":"2024-03-28T02:08:55.872868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:19:46.552447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872881+0000","last_change":"2024-03-28T02:08:49.255253+0000","last_active":"2024-03-28T02:08:55.872881+0000","last_peered":"2024-03-28T02:08:55.872881+0000","last_clean":"2024-03-28T02:08:55.872881+0000","last_became_active":"2024-03-28T02:08:49.254804+0000","last_became_peered":"2024-03-28T02:08:49.254804+0000","last_unstale":"2024-03-28T02:08:55.872881+0000","last_undegraded":"2024-03-28T02:08:55.872881+0000","last_fullsized":"2024-03-28T02:08:55.872881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:53:31.927830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293778+0000","last_change":"2024-03-28T02:08:53.497820+0000","last_active":"2024-03-28T02:08:55.293778+0000","last_peered":"2024-03-28T02:08:55.293778+0000","last_clean":"2024-03-28T02:08:55.293778+0000","last_became_active":"2024-03-28T02:08:53.497441+0000","last_became_peered":"2024-03-28T02:08:53.497441+0000","last_unstale":"2024-03-28T02:08:55.293778+0000","last_undegraded":"2024-03-28T02:08:55.293778+0000","last_fullsized":"2024-03-28T02:08:55.293778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:12:02.993121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873023+0000","last_change":"2024-03-28T02:08:48.445907+0000","last_active":"2024-03-28T02:08:55.873023+0000","last_peered":"2024-03-28T02:08:55.873023+0000","last_clean":"2024-03-28T02:08:55.873023+0000","last_became_active":"2024-03-28T02:08:48.444278+0000","last_became_peered":"2024-03-28T02:08:48.444278+0000","last_unstale":"2024-03-28T02:08:55.873023+0000","last_undegraded":"2024-03-28T02:08:55.873023+0000","last_fullsized":"2024-03-28T02:08:55.873023+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:23:30.060259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610584+0000","last_change":"2024-03-28T02:08:49.262862+0000","last_active":"2024-03-28T02:08:55.610584+0000","last_peered":"2024-03-28T02:08:55.610584+0000","last_clean":"2024-03-28T02:08:55.610584+0000","last_became_active":"2024-03-28T02:08:49.262451+0000","last_became_peered":"2024-03-28T02:08:49.262451+0000","last_unstale":"2024-03-28T02:08:55.610584+0000","last_undegraded":"2024-03-28T02:08:55.610584+0000","last_fullsized":"2024-03-28T02:08:55.610584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:12:51.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610891+0000","last_change":"2024-03-28T02:08:53.490855+0000","last_active":"2024-03-28T02:08:55.610891+0000","last_peered":"2024-03-28T02:08:55.610891+0000","last_clean":"2024-03-28T02:08:55.610891+0000","last_became_active":"2024-03-28T02:08:53.490415+0000","last_became_peered":"2024-03-28T02:08:53.490415+0000","last_unstale":"2024-03-28T02:08:55.610891+0000","last_undegraded":"2024-03-28T02:08:55.610891+0000","last_fullsized":"2024-03-28T02:08:55.610891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:56:37.426398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873783+0000","last_change":"2024-03-28T02:08:48.448909+0000","last_active":"2024-03-28T02:08:55.873783+0000","last_peered":"2024-03-28T02:08:55.873783+0000","last_clean":"2024-03-28T02:08:55.873783+0000","last_became_active":"2024-03-28T02:08:48.448548+0000","last_became_peered":"2024-03-28T02:08:48.448548+0000","last_unstale":"2024-03-28T02:08:55.873783+0000","last_undegraded":"2024-03-28T02:08:55.873783+0000","last_fullsized":"2024-03-28T02:08:55.873783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:51:06.206266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873065+0000","last_change":"2024-03-28T02:08:49.266920+0000","last_active":"2024-03-28T02:08:55.873065+0000","last_peered":"2024-03-28T02:08:55.873065+0000","last_clean":"2024-03-28T02:08:55.873065+0000","last_became_active":"2024-03-28T02:08:49.266600+0000","last_became_peered":"2024-03-28T02:08:49.266600+0000","last_unstale":"2024-03-28T02:08:55.873065+0000","last_undegraded":"2024-03-28T02:08:55.873065+0000","last_fullsized":"2024-03-28T02:08:55.873065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:57.735460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873183+0000","last_change":"2024-03-28T02:08:53.631636+0000","last_active":"2024-03-28T02:08:55.873183+0000","last_peered":"2024-03-28T02:08:55.873183+0000","last_clean":"2024-03-28T02:08:55.873183+0000","last_became_active":"2024-03-28T02:08:53.631183+0000","last_became_peered":"2024-03-28T02:08:53.631183+0000","last_unstale":"2024-03-28T02:08:55.873183+0000","last_undegraded":"2024-03-28T02:08:55.873183+0000","last_fullsized":"2024-03-28T02:08:55.873183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:22:09.212726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482576+0000","last_change":"2024-03-28T02:08:48.440789+0000","last_active":"2024-03-28T02:08:54.482576+0000","last_peered":"2024-03-28T02:08:54.482576+0000","last_clean":"2024-03-28T02:08:54.482576+0000","last_became_active":"2024-03-28T02:08:48.440403+0000","last_became_peered":"2024-03-28T02:08:48.440403+0000","last_unstale":"2024-03-28T02:08:54.482576+0000","last_undegraded":"2024-03-28T02:08:54.482576+0000","last_fullsized":"2024-03-28T02:08:54.482576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:37.227279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482918+0000","last_change":"2024-03-28T02:08:49.255340+0000","last_active":"2024-03-28T02:08:54.482918+0000","last_peered":"2024-03-28T02:08:54.482918+0000","last_clean":"2024-03-28T02:08:54.482918+0000","last_became_active":"2024-03-28T02:08:49.255027+0000","last_became_peered":"2024-03-28T02:08:49.255027+0000","last_unstale":"2024-03-28T02:08:54.482918+0000","last_undegraded":"2024-03-28T02:08:54.482918+0000","last_fullsized":"2024-03-28T02:08:54.482918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:54:03.454022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293642+0000","last_change":"2024-03-28T02:08:53.499721+0000","last_active":"2024-03-28T02:08:55.293642+0000","last_peered":"2024-03-28T02:08:55.293642+0000","last_clean":"2024-03-28T02:08:55.293642+0000","last_became_active":"2024-03-28T02:08:53.499117+0000","last_became_peered":"2024-03-28T02:08:53.499117+0000","last_unstale":"2024-03-28T02:08:55.293642+0000","last_undegraded":"2024-03-28T02:08:55.293642+0000","last_fullsized":"2024-03-28T02:08:55.293642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:07:33.561249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873088+0000","last_change":"2024-03-28T02:08:48.453592+0000","last_active":"2024-03-28T02:08:55.873088+0000","last_peered":"2024-03-28T02:08:55.873088+0000","last_clean":"2024-03-28T02:08:55.873088+0000","last_became_active":"2024-03-28T02:08:48.453169+0000","last_became_peered":"2024-03-28T02:08:48.453169+0000","last_unstale":"2024-03-28T02:08:55.873088+0000","last_undegraded":"2024-03-28T02:08:55.873088+0000","last_fullsized":"2024-03-28T02:08:55.873088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:33.163170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482700+0000","last_change":"2024-03-28T02:08:49.263143+0000","last_active":"2024-03-28T02:08:54.482700+0000","last_peered":"2024-03-28T02:08:54.482700+0000","last_clean":"2024-03-28T02:08:54.482700+0000","last_became_active":"2024-03-28T02:08:49.262829+0000","last_became_peered":"2024-03-28T02:08:49.262829+0000","last_unstale":"2024-03-28T02:08:54.482700+0000","last_undegraded":"2024-03-28T02:08:54.482700+0000","last_fullsized":"2024-03-28T02:08:54.482700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:01:07.809685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293785+0000","last_change":"2024-03-28T02:08:53.631595+0000","last_active":"2024-03-28T02:08:55.293785+0000","last_peered":"2024-03-28T02:08:55.293785+0000","last_clean":"2024-03-28T02:08:55.293785+0000","last_became_active":"2024-03-28T02:08:53.630863+0000","last_became_peered":"2024-03-28T02:08:53.630863+0000","last_unstale":"2024-03-28T02:08:55.293785+0000","last_undegraded":"2024-03-28T02:08:55.293785+0000","last_fullsized":"2024-03-28T02:08:55.293785+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:38:44.712728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610770+0000","last_change":"2024-03-28T02:08:49.263523+0000","last_active":"2024-03-28T02:08:55.610770+0000","last_peered":"2024-03-28T02:08:55.610770+0000","last_clean":"2024-03-28T02:08:55.610770+0000","last_became_active":"2024-03-28T02:08:49.263079+0000","last_became_peered":"2024-03-28T02:08:49.263079+0000","last_unstale":"2024-03-28T02:08:55.610770+0000","last_undegraded":"2024-03-28T02:08:55.610770+0000","last_fullsized":"2024-03-28T02:08:55.610770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:15:37.516032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293168+0000","last_change":"2024-03-28T02:08:48.455723+0000","last_active":"2024-03-28T02:08:55.293168+0000","last_peered":"2024-03-28T02:08:55.293168+0000","last_clean":"2024-03-28T02:08:55.293168+0000","last_became_active":"2024-03-28T02:08:48.455340+0000","last_became_peered":"2024-03-28T02:08:48.455340+0000","last_unstale":"2024-03-28T02:08:55.293168+0000","last_undegraded":"2024-03-28T02:08:55.293168+0000","last_fullsized":"2024-03-28T02:08:55.293168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:15:36.376221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873515+0000","last_change":"2024-03-28T02:08:53.486656+0000","last_active":"2024-03-28T02:08:55.873515+0000","last_peered":"2024-03-28T02:08:55.873515+0000","last_clean":"2024-03-28T02:08:55.873515+0000","last_became_active":"2024-03-28T02:08:53.486274+0000","last_became_peered":"2024-03-28T02:08:53.486274+0000","last_unstale":"2024-03-28T02:08:55.873515+0000","last_undegraded":"2024-03-28T02:08:55.873515+0000","last_fullsized":"2024-03-28T02:08:55.873515+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:40:40.742751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872651+0000","last_change":"2024-03-28T02:08:49.260380+0000","last_active":"2024-03-28T02:08:55.872651+0000","last_peered":"2024-03-28T02:08:55.872651+0000","last_clean":"2024-03-28T02:08:55.872651+0000","last_became_active":"2024-03-28T02:08:49.260012+0000","last_became_peered":"2024-03-28T02:08:49.260012+0000","last_unstale":"2024-03-28T02:08:55.872651+0000","last_undegraded":"2024-03-28T02:08:55.872651+0000","last_fullsized":"2024-03-28T02:08:55.872651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:46:31.233692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873406+0000","last_change":"2024-03-28T02:08:48.447659+0000","last_active":"2024-03-28T02:08:55.873406+0000","last_peered":"2024-03-28T02:08:55.873406+0000","last_clean":"2024-03-28T02:08:55.873406+0000","last_became_active":"2024-03-28T02:08:48.447263+0000","last_became_peered":"2024-03-28T02:08:48.447263+0000","last_unstale":"2024-03-28T02:08:55.873406+0000","last_undegraded":"2024-03-28T02:08:55.873406+0000","last_fullsized":"2024-03-28T02:08:55.873406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:16:32.453528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872752+0000","last_change":"2024-03-28T02:07:33.021769+0000","last_active":"2024-03-28T02:08:55.872752+0000","last_peered":"2024-03-28T02:08:55.872752+0000","last_clean":"2024-03-28T02:08:55.872752+0000","last_became_active":"2024-03-28T02:07:33.021338+0000","last_became_peered":"2024-03-28T02:07:33.021338+0000","last_unstale":"2024-03-28T02:08:55.872752+0000","last_undegraded":"2024-03-28T02:08:55.872752+0000","last_fullsized":"2024-03-28T02:08:55.872752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:04:07.152098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:04:07.152098+0000","last_clean_scrub_stamp":"2024-03-28T02:04:07.152098+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:01:39.981299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611268+0000","last_change":"2024-03-28T02:08:53.631687+0000","last_active":"2024-03-28T02:08:55.611268+0000","last_peered":"2024-03-28T02:08:55.611268+0000","last_clean":"2024-03-28T02:08:55.611268+0000","last_became_active":"2024-03-28T02:08:53.631291+0000","last_became_peered":"2024-03-28T02:08:53.631291+0000","last_unstale":"2024-03-28T02:08:55.611268+0000","last_undegraded":"2024-03-28T02:08:55.611268+0000","last_fullsized":"2024-03-28T02:08:55.611268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:18:51.563667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293163+0000","last_change":"2024-03-28T02:08:49.255310+0000","last_active":"2024-03-28T02:08:55.293163+0000","last_peered":"2024-03-28T02:08:55.293163+0000","last_clean":"2024-03-28T02:08:55.293163+0000","last_became_active":"2024-03-28T02:08:49.254923+0000","last_became_peered":"2024-03-28T02:08:49.254923+0000","last_unstale":"2024-03-28T02:08:55.293163+0000","last_undegraded":"2024-03-28T02:08:55.293163+0000","last_fullsized":"2024-03-28T02:08:55.293163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:34.660601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873419+0000","last_change":"2024-03-28T02:08:48.457646+0000","last_active":"2024-03-28T02:08:55.873419+0000","last_peered":"2024-03-28T02:08:55.873419+0000","last_clean":"2024-03-28T02:08:55.873419+0000","last_became_active":"2024-03-28T02:08:48.457344+0000","last_became_peered":"2024-03-28T02:08:48.457344+0000","last_unstale":"2024-03-28T02:08:55.873419+0000","last_undegraded":"2024-03-28T02:08:55.873419+0000","last_fullsized":"2024-03-28T02:08:55.873419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:25:19.077816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873072+0000","last_change":"2024-03-28T02:08:53.486320+0000","last_active":"2024-03-28T02:08:55.873072+0000","last_peered":"2024-03-28T02:08:55.873072+0000","last_clean":"2024-03-28T02:08:55.873072+0000","last_became_active":"2024-03-28T02:08:53.485771+0000","last_became_peered":"2024-03-28T02:08:53.485771+0000","last_unstale":"2024-03-28T02:08:55.873072+0000","last_undegraded":"2024-03-28T02:08:55.873072+0000","last_fullsized":"2024-03-28T02:08:55.873072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:48.452990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873555+0000","last_change":"2024-03-28T02:08:48.458854+0000","last_active":"2024-03-28T02:08:55.873555+0000","last_peered":"2024-03-28T02:08:55.873555+0000","last_clean":"2024-03-28T02:08:55.873555+0000","last_became_active":"2024-03-28T02:08:48.456667+0000","last_became_peered":"2024-03-28T02:08:48.456667+0000","last_unstale":"2024-03-28T02:08:55.873555+0000","last_undegraded":"2024-03-28T02:08:55.873555+0000","last_fullsized":"2024-03-28T02:08:55.873555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:37:54.113797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293659+0000","last_change":"2024-03-28T02:08:49.255300+0000","last_active":"2024-03-28T02:08:55.293659+0000","last_peered":"2024-03-28T02:08:55.293659+0000","last_clean":"2024-03-28T02:08:55.293659+0000","last_became_active":"2024-03-28T02:08:49.254955+0000","last_became_peered":"2024-03-28T02:08:49.254955+0000","last_unstale":"2024-03-28T02:08:55.293659+0000","last_undegraded":"2024-03-28T02:08:55.293659+0000","last_fullsized":"2024-03-28T02:08:55.293659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:29.093770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872623+0000","last_change":"2024-03-28T02:08:53.631966+0000","last_active":"2024-03-28T02:08:55.872623+0000","last_peered":"2024-03-28T02:08:55.872623+0000","last_clean":"2024-03-28T02:08:55.872623+0000","last_became_active":"2024-03-28T02:08:53.631457+0000","last_became_peered":"2024-03-28T02:08:53.631457+0000","last_unstale":"2024-03-28T02:08:55.872623+0000","last_undegraded":"2024-03-28T02:08:55.872623+0000","last_fullsized":"2024-03-28T02:08:55.872623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:36:46.623567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:08.258986+0000","last_change":"2024-03-28T02:08:48.452991+0000","last_active":"2024-03-28T02:09:08.258986+0000","last_peered":"2024-03-28T02:09:08.258986+0000","last_clean":"2024-03-28T02:09:08.258986+0000","last_became_active":"2024-03-28T02:08:48.452248+0000","last_became_peered":"2024-03-28T02:08:48.452248+0000","last_unstale":"2024-03-28T02:09:08.258986+0000","last_undegraded":"2024-03-28T02:09:08.258986+0000","last_fullsized":"2024-03-28T02:09:08.258986+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:13:40.100006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873065+0000","last_change":"2024-03-28T02:08:49.259575+0000","last_active":"2024-03-28T02:08:55.873065+0000","last_peered":"2024-03-28T02:08:55.873065+0000","last_clean":"2024-03-28T02:08:55.873065+0000","last_became_active":"2024-03-28T02:08:49.259255+0000","last_became_peered":"2024-03-28T02:08:49.259255+0000","last_unstale":"2024-03-28T02:08:55.873065+0000","last_undegraded":"2024-03-28T02:08:55.873065+0000","last_fullsized":"2024-03-28T02:08:55.873065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:47:17.741691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482818+0000","last_change":"2024-03-28T02:08:53.488304+0000","last_active":"2024-03-28T02:08:54.482818+0000","last_peered":"2024-03-28T02:08:54.482818+0000","last_clean":"2024-03-28T02:08:54.482818+0000","last_became_active":"2024-03-28T02:08:53.487783+0000","last_became_peered":"2024-03-28T02:08:53.487783+0000","last_unstale":"2024-03-28T02:08:54.482818+0000","last_undegraded":"2024-03-28T02:08:54.482818+0000","last_fullsized":"2024-03-28T02:08:54.482818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:54:50.494685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873080+0000","last_change":"2024-03-28T02:08:48.450795+0000","last_active":"2024-03-28T02:08:55.873080+0000","last_peered":"2024-03-28T02:08:55.873080+0000","last_clean":"2024-03-28T02:08:55.873080+0000","last_became_active":"2024-03-28T02:08:48.450449+0000","last_became_peered":"2024-03-28T02:08:48.450449+0000","last_unstale":"2024-03-28T02:08:55.873080+0000","last_undegraded":"2024-03-28T02:08:55.873080+0000","last_fullsized":"2024-03-28T02:08:55.873080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:44.576188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734045+0000","last_change":"2024-03-28T02:08:49.264220+0000","last_active":"2024-03-28T02:08:55.734045+0000","last_peered":"2024-03-28T02:08:55.734045+0000","last_clean":"2024-03-28T02:08:55.734045+0000","last_became_active":"2024-03-28T02:08:49.263757+0000","last_became_peered":"2024-03-28T02:08:49.263757+0000","last_unstale":"2024-03-28T02:08:55.734045+0000","last_undegraded":"2024-03-28T02:08:55.734045+0000","last_fullsized":"2024-03-28T02:08:55.734045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:20:44.000256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873280+0000","last_change":"2024-03-28T02:08:53.631904+0000","last_active":"2024-03-28T02:08:55.873280+0000","last_peered":"2024-03-28T02:08:55.873280+0000","last_clean":"2024-03-28T02:08:55.873280+0000","last_became_active":"2024-03-28T02:08:53.631263+0000","last_became_peered":"2024-03-28T02:08:53.631263+0000","last_unstale":"2024-03-28T02:08:55.873280+0000","last_undegraded":"2024-03-28T02:08:55.873280+0000","last_fullsized":"2024-03-28T02:08:55.873280+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:30:53.511632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760357+0000","last_change":"2024-03-28T02:08:48.450272+0000","last_active":"2024-03-28T02:08:54.760357+0000","last_peered":"2024-03-28T02:08:54.760357+0000","last_clean":"2024-03-28T02:08:54.760357+0000","last_became_active":"2024-03-28T02:08:48.449905+0000","last_became_peered":"2024-03-28T02:08:48.449905+0000","last_unstale":"2024-03-28T02:08:54.760357+0000","last_undegraded":"2024-03-28T02:08:54.760357+0000","last_fullsized":"2024-03-28T02:08:54.760357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:56:49.493569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873168+0000","last_change":"2024-03-28T02:08:49.256816+0000","last_active":"2024-03-28T02:08:55.873168+0000","last_peered":"2024-03-28T02:08:55.873168+0000","last_clean":"2024-03-28T02:08:55.873168+0000","last_became_active":"2024-03-28T02:08:49.256369+0000","last_became_peered":"2024-03-28T02:08:49.256369+0000","last_unstale":"2024-03-28T02:08:55.873168+0000","last_undegraded":"2024-03-28T02:08:55.873168+0000","last_fullsized":"2024-03-28T02:08:55.873168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:39.714619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482907+0000","last_change":"2024-03-28T02:08:53.496231+0000","last_active":"2024-03-28T02:08:54.482907+0000","last_peered":"2024-03-28T02:08:54.482907+0000","last_clean":"2024-03-28T02:08:54.482907+0000","last_became_active":"2024-03-28T02:08:53.494179+0000","last_became_peered":"2024-03-28T02:08:53.494179+0000","last_unstale":"2024-03-28T02:08:54.482907+0000","last_undegraded":"2024-03-28T02:08:54.482907+0000","last_fullsized":"2024-03-28T02:08:54.482907+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:17:17.231150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872575+0000","last_change":"2024-03-28T02:08:48.454998+0000","last_active":"2024-03-28T02:08:55.872575+0000","last_peered":"2024-03-28T02:08:55.872575+0000","last_clean":"2024-03-28T02:08:55.872575+0000","last_became_active":"2024-03-28T02:08:48.454674+0000","last_became_peered":"2024-03-28T02:08:48.454674+0000","last_unstale":"2024-03-28T02:08:55.872575+0000","last_undegraded":"2024-03-28T02:08:55.872575+0000","last_fullsized":"2024-03-28T02:08:55.872575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:01:23.432625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872754+0000","last_change":"2024-03-28T02:08:49.263714+0000","last_active":"2024-03-28T02:08:55.872754+0000","last_peered":"2024-03-28T02:08:55.872754+0000","last_clean":"2024-03-28T02:08:55.872754+0000","last_became_active":"2024-03-28T02:08:49.263335+0000","last_became_peered":"2024-03-28T02:08:49.263335+0000","last_unstale":"2024-03-28T02:08:55.872754+0000","last_undegraded":"2024-03-28T02:08:55.872754+0000","last_fullsized":"2024-03-28T02:08:55.872754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:28:13.153319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873827+0000","last_change":"2024-03-28T02:08:53.496751+0000","last_active":"2024-03-28T02:08:55.873827+0000","last_peered":"2024-03-28T02:08:55.873827+0000","last_clean":"2024-03-28T02:08:55.873827+0000","last_became_active":"2024-03-28T02:08:53.496393+0000","last_became_peered":"2024-03-28T02:08:53.496393+0000","last_unstale":"2024-03-28T02:08:55.873827+0000","last_undegraded":"2024-03-28T02:08:55.873827+0000","last_fullsized":"2024-03-28T02:08:55.873827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:44:13.268278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482840+0000","last_change":"2024-03-28T02:08:48.456026+0000","last_active":"2024-03-28T02:08:54.482840+0000","last_peered":"2024-03-28T02:08:54.482840+0000","last_clean":"2024-03-28T02:08:54.482840+0000","last_became_active":"2024-03-28T02:08:48.455264+0000","last_became_peered":"2024-03-28T02:08:48.455264+0000","last_unstale":"2024-03-28T02:08:54.482840+0000","last_undegraded":"2024-03-28T02:08:54.482840+0000","last_fullsized":"2024-03-28T02:08:54.482840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:21:51.682413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872934+0000","last_change":"2024-03-28T02:08:49.260416+0000","last_active":"2024-03-28T02:08:55.872934+0000","last_peered":"2024-03-28T02:08:55.872934+0000","last_clean":"2024-03-28T02:08:55.872934+0000","last_became_active":"2024-03-28T02:08:49.260071+0000","last_became_peered":"2024-03-28T02:08:49.260071+0000","last_unstale":"2024-03-28T02:08:55.872934+0000","last_undegraded":"2024-03-28T02:08:55.872934+0000","last_fullsized":"2024-03-28T02:08:55.872934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:34:56.414305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873021+0000","last_change":"2024-03-28T02:08:53.631633+0000","last_active":"2024-03-28T02:08:55.873021+0000","last_peered":"2024-03-28T02:08:55.873021+0000","last_clean":"2024-03-28T02:08:55.873021+0000","last_became_active":"2024-03-28T02:08:53.631158+0000","last_became_peered":"2024-03-28T02:08:53.631158+0000","last_unstale":"2024-03-28T02:08:55.873021+0000","last_undegraded":"2024-03-28T02:08:55.873021+0000","last_fullsized":"2024-03-28T02:08:55.873021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:37:43.749860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873453+0000","last_change":"2024-03-28T02:08:48.454468+0000","last_active":"2024-03-28T02:08:55.873453+0000","last_peered":"2024-03-28T02:08:55.873453+0000","last_clean":"2024-03-28T02:08:55.873453+0000","last_became_active":"2024-03-28T02:08:48.454086+0000","last_became_peered":"2024-03-28T02:08:48.454086+0000","last_unstale":"2024-03-28T02:08:55.873453+0000","last_undegraded":"2024-03-28T02:08:55.873453+0000","last_fullsized":"2024-03-28T02:08:55.873453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:03.178254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873118+0000","last_change":"2024-03-28T02:08:49.268529+0000","last_active":"2024-03-28T02:08:55.873118+0000","last_peered":"2024-03-28T02:08:55.873118+0000","last_clean":"2024-03-28T02:08:55.873118+0000","last_became_active":"2024-03-28T02:08:49.267869+0000","last_became_peered":"2024-03-28T02:08:49.267869+0000","last_unstale":"2024-03-28T02:08:55.873118+0000","last_undegraded":"2024-03-28T02:08:55.873118+0000","last_fullsized":"2024-03-28T02:08:55.873118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:52:34.633616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760699+0000","last_change":"2024-03-28T02:08:53.495894+0000","last_active":"2024-03-28T02:08:54.760699+0000","last_peered":"2024-03-28T02:08:54.760699+0000","last_clean":"2024-03-28T02:08:54.760699+0000","last_became_active":"2024-03-28T02:08:53.495442+0000","last_became_peered":"2024-03-28T02:08:53.495442+0000","last_unstale":"2024-03-28T02:08:54.760699+0000","last_undegraded":"2024-03-28T02:08:54.760699+0000","last_fullsized":"2024-03-28T02:08:54.760699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:32:33.318061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483338+0000","last_change":"2024-03-28T02:08:48.450376+0000","last_active":"2024-03-28T02:08:54.483338+0000","last_peered":"2024-03-28T02:08:54.483338+0000","last_clean":"2024-03-28T02:08:54.483338+0000","last_became_active":"2024-03-28T02:08:48.449978+0000","last_became_peered":"2024-03-28T02:08:48.449978+0000","last_unstale":"2024-03-28T02:08:54.483338+0000","last_undegraded":"2024-03-28T02:08:54.483338+0000","last_fullsized":"2024-03-28T02:08:54.483338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:33:56.284025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873674+0000","last_change":"2024-03-28T02:08:49.269011+0000","last_active":"2024-03-28T02:08:55.873674+0000","last_peered":"2024-03-28T02:08:55.873674+0000","last_clean":"2024-03-28T02:08:55.873674+0000","last_became_active":"2024-03-28T02:08:49.268141+0000","last_became_peered":"2024-03-28T02:08:49.268141+0000","last_unstale":"2024-03-28T02:08:55.873674+0000","last_undegraded":"2024-03-28T02:08:55.873674+0000","last_fullsized":"2024-03-28T02:08:55.873674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:54:32.708445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.733856+0000","last_change":"2024-03-28T02:08:53.501549+0000","last_active":"2024-03-28T02:08:55.733856+0000","last_peered":"2024-03-28T02:08:55.733856+0000","last_clean":"2024-03-28T02:08:55.733856+0000","last_became_active":"2024-03-28T02:08:53.501017+0000","last_became_peered":"2024-03-28T02:08:53.501017+0000","last_unstale":"2024-03-28T02:08:55.733856+0000","last_undegraded":"2024-03-28T02:08:55.733856+0000","last_fullsized":"2024-03-28T02:08:55.733856+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:33:50.346117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611029+0000","last_change":"2024-03-28T02:08:48.454671+0000","last_active":"2024-03-28T02:08:55.611029+0000","last_peered":"2024-03-28T02:08:55.611029+0000","last_clean":"2024-03-28T02:08:55.611029+0000","last_became_active":"2024-03-28T02:08:48.454329+0000","last_became_peered":"2024-03-28T02:08:48.454329+0000","last_unstale":"2024-03-28T02:08:55.611029+0000","last_undegraded":"2024-03-28T02:08:55.611029+0000","last_fullsized":"2024-03-28T02:08:55.611029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:41:24.087559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873169+0000","last_change":"2024-03-28T02:08:49.259510+0000","last_active":"2024-03-28T02:08:55.873169+0000","last_peered":"2024-03-28T02:08:55.873169+0000","last_clean":"2024-03-28T02:08:55.873169+0000","last_became_active":"2024-03-28T02:08:49.259141+0000","last_became_peered":"2024-03-28T02:08:49.259141+0000","last_unstale":"2024-03-28T02:08:55.873169+0000","last_undegraded":"2024-03-28T02:08:55.873169+0000","last_fullsized":"2024-03-28T02:08:55.873169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:56:50.754576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610477+0000","last_change":"2024-03-28T02:08:53.488792+0000","last_active":"2024-03-28T02:08:55.610477+0000","last_peered":"2024-03-28T02:08:55.610477+0000","last_clean":"2024-03-28T02:08:55.610477+0000","last_became_active":"2024-03-28T02:08:53.488013+0000","last_became_peered":"2024-03-28T02:08:53.488013+0000","last_unstale":"2024-03-28T02:08:55.610477+0000","last_undegraded":"2024-03-28T02:08:55.610477+0000","last_fullsized":"2024-03-28T02:08:55.610477+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:39:52.006618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293634+0000","last_change":"2024-03-28T02:08:48.454609+0000","last_active":"2024-03-28T02:08:55.293634+0000","last_peered":"2024-03-28T02:08:55.293634+0000","last_clean":"2024-03-28T02:08:55.293634+0000","last_became_active":"2024-03-28T02:08:48.454297+0000","last_became_peered":"2024-03-28T02:08:48.454297+0000","last_unstale":"2024-03-28T02:08:55.293634+0000","last_undegraded":"2024-03-28T02:08:55.293634+0000","last_fullsized":"2024-03-28T02:08:55.293634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:31:42.819338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873773+0000","last_change":"2024-03-28T02:08:49.263700+0000","last_active":"2024-03-28T02:08:55.873773+0000","last_peered":"2024-03-28T02:08:55.873773+0000","last_clean":"2024-03-28T02:08:55.873773+0000","last_became_active":"2024-03-28T02:08:49.263073+0000","last_became_peered":"2024-03-28T02:08:49.263073+0000","last_unstale":"2024-03-28T02:08:55.873773+0000","last_undegraded":"2024-03-28T02:08:55.873773+0000","last_fullsized":"2024-03-28T02:08:55.873773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:34:19.158284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872835+0000","last_change":"2024-03-28T02:08:49.265572+0000","last_active":"2024-03-28T02:08:55.872835+0000","last_peered":"2024-03-28T02:08:55.872835+0000","last_clean":"2024-03-28T02:08:55.872835+0000","last_became_active":"2024-03-28T02:08:49.265202+0000","last_became_peered":"2024-03-28T02:08:49.265202+0000","last_unstale":"2024-03-28T02:08:55.872835+0000","last_undegraded":"2024-03-28T02:08:55.872835+0000","last_fullsized":"2024-03-28T02:08:55.872835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:00:05.296762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482974+0000","last_change":"2024-03-28T02:08:49.266529+0000","last_active":"2024-03-28T02:08:54.482974+0000","last_peered":"2024-03-28T02:08:54.482974+0000","last_clean":"2024-03-28T02:08:54.482974+0000","last_became_active":"2024-03-28T02:08:49.266162+0000","last_became_peered":"2024-03-28T02:08:49.266162+0000","last_unstale":"2024-03-28T02:08:54.482974+0000","last_undegraded":"2024-03-28T02:08:54.482974+0000","last_fullsized":"2024-03-28T02:08:54.482974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:31:40.117845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482760+0000","last_change":"2024-03-28T02:08:49.264712+0000","last_active":"2024-03-28T02:08:54.482760+0000","last_peered":"2024-03-28T02:08:54.482760+0000","last_clean":"2024-03-28T02:08:54.482760+0000","last_became_active":"2024-03-28T02:08:49.264363+0000","last_became_peered":"2024-03-28T02:08:49.264363+0000","last_unstale":"2024-03-28T02:08:54.482760+0000","last_undegraded":"2024-03-28T02:08:54.482760+0000","last_fullsized":"2024-03-28T02:08:54.482760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:14:09.178725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734001+0000","last_change":"2024-03-28T02:08:49.261713+0000","last_active":"2024-03-28T02:08:55.734001+0000","last_peered":"2024-03-28T02:08:55.734001+0000","last_clean":"2024-03-28T02:08:55.734001+0000","last_became_active":"2024-03-28T02:08:49.261311+0000","last_became_peered":"2024-03-28T02:08:49.261311+0000","last_unstale":"2024-03-28T02:08:55.734001+0000","last_undegraded":"2024-03-28T02:08:55.734001+0000","last_fullsized":"2024-03-28T02:08:55.734001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:56:54.801575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293500+0000","last_change":"2024-03-28T02:08:49.256674+0000","last_active":"2024-03-28T02:08:55.293500+0000","last_peered":"2024-03-28T02:08:55.293500+0000","last_clean":"2024-03-28T02:08:55.293500+0000","last_became_active":"2024-03-28T02:08:49.256360+0000","last_became_peered":"2024-03-28T02:08:49.256360+0000","last_unstale":"2024-03-28T02:08:55.293500+0000","last_undegraded":"2024-03-28T02:08:55.293500+0000","last_fullsized":"2024-03-28T02:08:55.293500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:44:14.817569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873468+0000","last_change":"2024-03-28T02:08:49.259829+0000","last_active":"2024-03-28T02:08:55.873468+0000","last_peered":"2024-03-28T02:08:55.873468+0000","last_clean":"2024-03-28T02:08:55.873468+0000","last_became_active":"2024-03-28T02:08:49.259419+0000","last_became_peered":"2024-03-28T02:08:49.259419+0000","last_unstale":"2024-03-28T02:08:55.873468+0000","last_undegraded":"2024-03-28T02:08:55.873468+0000","last_fullsized":"2024-03-28T02:08:55.873468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:49:46.023075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483527+0000","last_change":"2024-03-28T02:08:49.263116+0000","last_active":"2024-03-28T02:08:54.483527+0000","last_peered":"2024-03-28T02:08:54.483527+0000","last_clean":"2024-03-28T02:08:54.483527+0000","last_became_active":"2024-03-28T02:08:49.262568+0000","last_became_peered":"2024-03-28T02:08:49.262568+0000","last_unstale":"2024-03-28T02:08:54.483527+0000","last_undegraded":"2024-03-28T02:08:54.483527+0000","last_fullsized":"2024-03-28T02:08:54.483527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:29:50.397737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873142+0000","last_change":"2024-03-28T02:08:49.261449+0000","last_active":"2024-03-28T02:08:55.873142+0000","last_peered":"2024-03-28T02:08:55.873142+0000","last_clean":"2024-03-28T02:08:55.873142+0000","last_became_active":"2024-03-28T02:08:49.261092+0000","last_became_peered":"2024-03-28T02:08:49.261092+0000","last_unstale":"2024-03-28T02:08:55.873142+0000","last_undegraded":"2024-03-28T02:08:55.873142+0000","last_fullsized":"2024-03-28T02:08:55.873142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:51:39.388852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610776+0000","last_change":"2024-03-28T02:08:49.260711+0000","last_active":"2024-03-28T02:08:55.610776+0000","last_peered":"2024-03-28T02:08:55.610776+0000","last_clean":"2024-03-28T02:08:55.610776+0000","last_became_active":"2024-03-28T02:08:49.259358+0000","last_became_peered":"2024-03-28T02:08:49.259358+0000","last_unstale":"2024-03-28T02:08:55.610776+0000","last_undegraded":"2024-03-28T02:08:55.610776+0000","last_fullsized":"2024-03-28T02:08:55.610776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:27:09.084143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872568+0000","last_change":"2024-03-28T02:08:49.267145+0000","last_active":"2024-03-28T02:08:55.872568+0000","last_peered":"2024-03-28T02:08:55.872568+0000","last_clean":"2024-03-28T02:08:55.872568+0000","last_became_active":"2024-03-28T02:08:49.266826+0000","last_became_peered":"2024-03-28T02:08:49.266826+0000","last_unstale":"2024-03-28T02:08:55.872568+0000","last_undegraded":"2024-03-28T02:08:55.872568+0000","last_fullsized":"2024-03-28T02:08:55.872568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:37:20.143596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760661+0000","last_change":"2024-03-28T02:08:49.266752+0000","last_active":"2024-03-28T02:08:54.760661+0000","last_peered":"2024-03-28T02:08:54.760661+0000","last_clean":"2024-03-28T02:08:54.760661+0000","last_became_active":"2024-03-28T02:08:49.266436+0000","last_became_peered":"2024-03-28T02:08:49.266436+0000","last_unstale":"2024-03-28T02:08:54.760661+0000","last_undegraded":"2024-03-28T02:08:54.760661+0000","last_fullsized":"2024-03-28T02:08:54.760661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:52:26.578174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293471+0000","last_change":"2024-03-28T02:08:49.258934+0000","last_active":"2024-03-28T02:08:55.293471+0000","last_peered":"2024-03-28T02:08:55.293471+0000","last_clean":"2024-03-28T02:08:55.293471+0000","last_became_active":"2024-03-28T02:08:49.258403+0000","last_became_peered":"2024-03-28T02:08:49.258403+0000","last_unstale":"2024-03-28T02:08:55.293471+0000","last_undegraded":"2024-03-28T02:08:55.293471+0000","last_fullsized":"2024-03-28T02:08:55.293471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:44:45.886136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873307+0000","last_change":"2024-03-28T02:08:49.268456+0000","last_active":"2024-03-28T02:08:55.873307+0000","last_peered":"2024-03-28T02:08:55.873307+0000","last_clean":"2024-03-28T02:08:55.873307+0000","last_became_active":"2024-03-28T02:08:49.267346+0000","last_became_peered":"2024-03-28T02:08:49.267346+0000","last_unstale":"2024-03-28T02:08:55.873307+0000","last_undegraded":"2024-03-28T02:08:55.873307+0000","last_fullsized":"2024-03-28T02:08:55.873307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:16:47.157938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283904,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4730,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":68,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":16928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57,"ondisk_log_size":57,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1073003,"data_compressed":50774,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":477819,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":50774,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477909,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50774,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T02:09:11.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T02:09:12.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.126+0000 7f5674174640 1 -- 172.21.15.106:0/899967051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566c1073a0 msgr2=0x7f566c1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.126+0000 7f5674174640 1 --2- 172.21.15.106:0/899967051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566c1073a0 0x7f566c1097f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5664009ca0 tx=0x7f566402c0f0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.126+0000 7f5674174640 1 -- 172.21.15.106:0/899967051 shutdown_connections 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.126+0000 7f5674174640 1 --2- 172.21.15.106:0/899967051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566c1073a0 0x7f566c1097f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.126+0000 7f5674174640 1 --2- 172.21.15.106:0/899967051 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566c104a70 0x7f566c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.126+0000 7f5674174640 1 --2- 172.21.15.106:0/899967051 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f566c102140 0x7f566c104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.126+0000 7f5674174640 1 -- 172.21.15.106:0/899967051 >> 172.21.15.106:0/899967051 conn(0x7f566c0fbb30 msgr2=0x7f566c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.126+0000 7f5674174640 1 -- 172.21.15.106:0/899967051 shutdown_connections 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.126+0000 7f5674174640 1 -- 172.21.15.106:0/899967051 wait complete. 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5674174640 1 Processor -- start 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5674174640 1 -- start start 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5674174640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566c102140 0x7f566c197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5674174640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566c104a70 0x7f566c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5674174640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f566c1073a0 0x7f566c19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5674174640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c10d340 con 0x7f566c104a70 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5674174640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c10d1c0 con 0x7f566c102140 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5674174640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c10d4c0 con 0x7f566c1073a0 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f567170f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566c104a70 0x7f566c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f567170f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566c104a70 0x7f566c197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35862/0 (socket says 172.21.15.106:35862) 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f567170f640 1 -- 172.21.15.106:0/2676906334 learned_addr learned my addr 172.21.15.106:0/2676906334 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5671f10640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566c102140 0x7f566c197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f5672711640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f566c1073a0 0x7f566c19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f567170f640 1 -- 172.21.15.106:0/2676906334 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f566c1073a0 msgr2=0x7f566c19ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f567170f640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f566c1073a0 0x7f566c19ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f567170f640 1 -- 172.21.15.106:0/2676906334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566c102140 msgr2=0x7f566c197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f567170f640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566c102140 0x7f566c197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f567170f640 1 -- 172.21.15.106:0/2676906334 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c19f570 con 0x7f566c104a70 2024-03-28T02:09:12.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.127+0000 7f567170f640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566c104a70 0x7f566c197e90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f565c00e9c0 tx=0x7f565c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:12.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.128+0000 7f5662ffd640 1 -- 172.21.15.106:0/2676906334 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c00cd80 con 0x7f566c104a70 2024-03-28T02:09:12.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.128+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c19f860 con 0x7f566c104a70 2024-03-28T02:09:12.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.128+0000 7f5662ffd640 1 -- 172.21.15.106:0/2676906334 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f565c005500 con 0x7f566c104a70 2024-03-28T02:09:12.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.128+0000 7f5662ffd640 1 -- 172.21.15.106:0/2676906334 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c010690 con 0x7f566c104a70 2024-03-28T02:09:12.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.128+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c19fe00 con 0x7f566c104a70 2024-03-28T02:09:12.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.129+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f566c0fd620 con 0x7f566c104a70 2024-03-28T02:09:12.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.130+0000 7f5662ffd640 1 -- 172.21.15.106:0/2676906334 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f565c0046c0 con 0x7f566c104a70 2024-03-28T02:09:12.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.130+0000 7f5662ffd640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56580a2890 0x7f56580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:12.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.130+0000 7f5671f10640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56580a2890 0x7f56580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:12.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.130+0000 7f5662ffd640 1 -- 172.21.15.106:0/2676906334 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f565c0cf190 con 0x7f566c104a70 2024-03-28T02:09:12.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.131+0000 7f5671f10640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56580a2890 0x7f56580a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f56680095f0 tx=0x7f5668005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:12.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.135+0000 7f5662ffd640 1 -- 172.21.15.106:0/2676906334 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f565c09be90 con 0x7f566c104a70 2024-03-28T02:09:12.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.310+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f566c061ae0 con 0x7f56580a2890 2024-03-28T02:09:12.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.319+0000 7f5662ffd640 1 -- 172.21.15.106:0/2676906334 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340500 (secure 0 0 0) 0x7f566c061ae0 con 0x7f56580a2890 2024-03-28T02:09:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:12.317 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56580a2890 msgr2=0x7f56580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56580a2890 0x7f56580a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f56680095f0 tx=0x7f5668005f90 comp rx=0 tx=0).stop 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566c104a70 msgr2=0x7f566c197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566c104a70 0x7f566c197e90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f565c00e9c0 tx=0x7f565c00ee80 comp rx=0 tx=0).stop 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 shutdown_connections 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f566c1073a0 0x7f566c19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566c104a70 0x7f566c197e90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56580a2890 0x7f56580a4d50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 --2- 172.21.15.106:0/2676906334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566c102140 0x7f566c197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 >> 172.21.15.106:0/2676906334 conn(0x7f566c0fbb30 msgr2=0x7f566c102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:12.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 shutdown_connections 2024-03-28T02:09:12.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:12.322+0000 7f5674174640 1 -- 172.21.15.106:0/2676906334 wait complete. 2024-03-28T02:09:12.327 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-28T02:09:10.355899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595098,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":142,"num_read_kb":127,"num_write":249,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377380,"kb_used_data":11548,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564444,"statfs":{"total":1151940427776,"available":1151553990656,"internally_reserved":0,"allocated":11825152,"data_stored":7513592,"data_compressed":284361,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2320,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":23,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003188"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873745+0000","last_change":"2024-03-28T02:08:49.268140+0000","last_active":"2024-03-28T02:08:55.873745+0000","last_peered":"2024-03-28T02:08:55.873745+0000","last_clean":"2024-03-28T02:08:55.873745+0000","last_became_active":"2024-03-28T02:08:49.267695+0000","last_became_peered":"2024-03-28T02:08:49.267695+0000","last_unstale":"2024-03-28T02:08:55.873745+0000","last_undegraded":"2024-03-28T02:08:55.873745+0000","last_fullsized":"2024-03-28T02:08:55.873745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:27:09.768237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734162+0000","last_change":"2024-03-28T02:08:49.259914+0000","last_active":"2024-03-28T02:08:55.734162+0000","last_peered":"2024-03-28T02:08:55.734162+0000","last_clean":"2024-03-28T02:08:55.734162+0000","last_became_active":"2024-03-28T02:08:49.259579+0000","last_became_peered":"2024-03-28T02:08:49.259579+0000","last_unstale":"2024-03-28T02:08:55.734162+0000","last_undegraded":"2024-03-28T02:08:55.734162+0000","last_fullsized":"2024-03-28T02:08:55.734162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:04:02.664429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734021+0000","last_change":"2024-03-28T02:08:49.268234+0000","last_active":"2024-03-28T02:08:55.734021+0000","last_peered":"2024-03-28T02:08:55.734021+0000","last_clean":"2024-03-28T02:08:55.734021+0000","last_became_active":"2024-03-28T02:08:49.267813+0000","last_became_peered":"2024-03-28T02:08:49.267813+0000","last_unstale":"2024-03-28T02:08:55.734021+0000","last_undegraded":"2024-03-28T02:08:55.734021+0000","last_fullsized":"2024-03-28T02:08:55.734021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:46:21.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873447+0000","last_change":"2024-03-28T02:08:49.268397+0000","last_active":"2024-03-28T02:08:55.873447+0000","last_peered":"2024-03-28T02:08:55.873447+0000","last_clean":"2024-03-28T02:08:55.873447+0000","last_became_active":"2024-03-28T02:08:49.268059+0000","last_became_peered":"2024-03-28T02:08:49.268059+0000","last_unstale":"2024-03-28T02:08:55.873447+0000","last_undegraded":"2024-03-28T02:08:55.873447+0000","last_fullsized":"2024-03-28T02:08:55.873447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:45:51.951306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873348+0000","last_change":"2024-03-28T02:08:49.257827+0000","last_active":"2024-03-28T02:08:55.873348+0000","last_peered":"2024-03-28T02:08:55.873348+0000","last_clean":"2024-03-28T02:08:55.873348+0000","last_became_active":"2024-03-28T02:08:49.256707+0000","last_became_peered":"2024-03-28T02:08:49.256707+0000","last_unstale":"2024-03-28T02:08:55.873348+0000","last_undegraded":"2024-03-28T02:08:55.873348+0000","last_fullsized":"2024-03-28T02:08:55.873348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:23:13.617900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482650+0000","last_change":"2024-03-28T02:08:49.262545+0000","last_active":"2024-03-28T02:08:54.482650+0000","last_peered":"2024-03-28T02:08:54.482650+0000","last_clean":"2024-03-28T02:08:54.482650+0000","last_became_active":"2024-03-28T02:08:49.262218+0000","last_became_peered":"2024-03-28T02:08:49.262218+0000","last_unstale":"2024-03-28T02:08:54.482650+0000","last_undegraded":"2024-03-28T02:08:54.482650+0000","last_fullsized":"2024-03-28T02:08:54.482650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:55:20.418608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873302+0000","last_change":"2024-03-28T02:08:49.260072+0000","last_active":"2024-03-28T02:08:55.873302+0000","last_peered":"2024-03-28T02:08:55.873302+0000","last_clean":"2024-03-28T02:08:55.873302+0000","last_became_active":"2024-03-28T02:08:49.259724+0000","last_became_peered":"2024-03-28T02:08:49.259724+0000","last_unstale":"2024-03-28T02:08:55.873302+0000","last_undegraded":"2024-03-28T02:08:55.873302+0000","last_fullsized":"2024-03-28T02:08:55.873302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:05.445467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872432+0000","last_change":"2024-03-28T02:08:49.262547+0000","last_active":"2024-03-28T02:08:55.872432+0000","last_peered":"2024-03-28T02:08:55.872432+0000","last_clean":"2024-03-28T02:08:55.872432+0000","last_became_active":"2024-03-28T02:08:49.261836+0000","last_became_peered":"2024-03-28T02:08:49.261836+0000","last_unstale":"2024-03-28T02:08:55.872432+0000","last_undegraded":"2024-03-28T02:08:55.872432+0000","last_fullsized":"2024-03-28T02:08:55.872432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:55:22.201524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760658+0000","last_change":"2024-03-28T02:08:49.262725+0000","last_active":"2024-03-28T02:08:54.760658+0000","last_peered":"2024-03-28T02:08:54.760658+0000","last_clean":"2024-03-28T02:08:54.760658+0000","last_became_active":"2024-03-28T02:08:49.262319+0000","last_became_peered":"2024-03-28T02:08:49.262319+0000","last_unstale":"2024-03-28T02:08:54.760658+0000","last_undegraded":"2024-03-28T02:08:54.760658+0000","last_fullsized":"2024-03-28T02:08:54.760658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:56:24.904009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483037+0000","last_change":"2024-03-28T02:08:49.262194+0000","last_active":"2024-03-28T02:08:54.483037+0000","last_peered":"2024-03-28T02:08:54.483037+0000","last_clean":"2024-03-28T02:08:54.483037+0000","last_became_active":"2024-03-28T02:08:49.261887+0000","last_became_peered":"2024-03-28T02:08:49.261887+0000","last_unstale":"2024-03-28T02:08:54.483037+0000","last_undegraded":"2024-03-28T02:08:54.483037+0000","last_fullsized":"2024-03-28T02:08:54.483037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:52:47.049430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873632+0000","last_change":"2024-03-28T02:08:49.269222+0000","last_active":"2024-03-28T02:08:55.873632+0000","last_peered":"2024-03-28T02:08:55.873632+0000","last_clean":"2024-03-28T02:08:55.873632+0000","last_became_active":"2024-03-28T02:08:49.268647+0000","last_became_peered":"2024-03-28T02:08:49.268647+0000","last_unstale":"2024-03-28T02:08:55.873632+0000","last_undegraded":"2024-03-28T02:08:55.873632+0000","last_fullsized":"2024-03-28T02:08:55.873632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:10:01.108158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873345+0000","last_change":"2024-03-28T02:08:49.269868+0000","last_active":"2024-03-28T02:08:55.873345+0000","last_peered":"2024-03-28T02:08:55.873345+0000","last_clean":"2024-03-28T02:08:55.873345+0000","last_became_active":"2024-03-28T02:08:49.269035+0000","last_became_peered":"2024-03-28T02:08:49.269035+0000","last_unstale":"2024-03-28T02:08:55.873345+0000","last_undegraded":"2024-03-28T02:08:55.873345+0000","last_fullsized":"2024-03-28T02:08:55.873345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:31.153970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293465+0000","last_change":"2024-03-28T02:08:49.254162+0000","last_active":"2024-03-28T02:08:55.293465+0000","last_peered":"2024-03-28T02:08:55.293465+0000","last_clean":"2024-03-28T02:08:55.293465+0000","last_became_active":"2024-03-28T02:08:49.253756+0000","last_became_peered":"2024-03-28T02:08:49.253756+0000","last_unstale":"2024-03-28T02:08:55.293465+0000","last_undegraded":"2024-03-28T02:08:55.293465+0000","last_fullsized":"2024-03-28T02:08:55.293465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:22:20.791809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.733751+0000","last_change":"2024-03-28T02:08:49.257940+0000","last_active":"2024-03-28T02:08:55.733751+0000","last_peered":"2024-03-28T02:08:55.733751+0000","last_clean":"2024-03-28T02:08:55.733751+0000","last_became_active":"2024-03-28T02:08:49.257039+0000","last_became_peered":"2024-03-28T02:08:49.257039+0000","last_unstale":"2024-03-28T02:08:55.733751+0000","last_undegraded":"2024-03-28T02:08:55.733751+0000","last_fullsized":"2024-03-28T02:08:55.733751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:36:10.701760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.733744+0000","last_change":"2024-03-28T02:08:49.267502+0000","last_active":"2024-03-28T02:08:55.733744+0000","last_peered":"2024-03-28T02:08:55.733744+0000","last_clean":"2024-03-28T02:08:55.733744+0000","last_became_active":"2024-03-28T02:08:49.265447+0000","last_became_peered":"2024-03-28T02:08:49.265447+0000","last_unstale":"2024-03-28T02:08:55.733744+0000","last_undegraded":"2024-03-28T02:08:55.733744+0000","last_fullsized":"2024-03-28T02:08:55.733744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:47:44.891440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760339+0000","last_change":"2024-03-28T02:08:49.267702+0000","last_active":"2024-03-28T02:08:54.760339+0000","last_peered":"2024-03-28T02:08:54.760339+0000","last_clean":"2024-03-28T02:08:54.760339+0000","last_became_active":"2024-03-28T02:08:49.267381+0000","last_became_peered":"2024-03-28T02:08:49.267381+0000","last_unstale":"2024-03-28T02:08:54.760339+0000","last_undegraded":"2024-03-28T02:08:54.760339+0000","last_fullsized":"2024-03-28T02:08:54.760339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:55:59.904141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873564+0000","last_change":"2024-03-28T02:08:49.268240+0000","last_active":"2024-03-28T02:08:55.873564+0000","last_peered":"2024-03-28T02:08:55.873564+0000","last_clean":"2024-03-28T02:08:55.873564+0000","last_became_active":"2024-03-28T02:08:49.267884+0000","last_became_peered":"2024-03-28T02:08:49.267884+0000","last_unstale":"2024-03-28T02:08:55.873564+0000","last_undegraded":"2024-03-28T02:08:55.873564+0000","last_fullsized":"2024-03-28T02:08:55.873564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:34:23.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482698+0000","last_change":"2024-03-28T02:08:49.262585+0000","last_active":"2024-03-28T02:08:54.482698+0000","last_peered":"2024-03-28T02:08:54.482698+0000","last_clean":"2024-03-28T02:08:54.482698+0000","last_became_active":"2024-03-28T02:08:49.262217+0000","last_became_peered":"2024-03-28T02:08:49.262217+0000","last_unstale":"2024-03-28T02:08:54.482698+0000","last_undegraded":"2024-03-28T02:08:54.482698+0000","last_fullsized":"2024-03-28T02:08:54.482698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:10:05.714733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483506+0000","last_change":"2024-03-28T02:08:49.261550+0000","last_active":"2024-03-28T02:08:54.483506+0000","last_peered":"2024-03-28T02:08:54.483506+0000","last_clean":"2024-03-28T02:08:54.483506+0000","last_became_active":"2024-03-28T02:08:49.261196+0000","last_became_peered":"2024-03-28T02:08:49.261196+0000","last_unstale":"2024-03-28T02:08:54.483506+0000","last_undegraded":"2024-03-28T02:08:54.483506+0000","last_fullsized":"2024-03-28T02:08:54.483506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:52:52.972048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483056+0000","last_change":"2024-03-28T02:08:53.491877+0000","last_active":"2024-03-28T02:08:54.483056+0000","last_peered":"2024-03-28T02:08:54.483056+0000","last_clean":"2024-03-28T02:08:54.483056+0000","last_became_active":"2024-03-28T02:08:53.491584+0000","last_became_peered":"2024-03-28T02:08:53.491584+0000","last_unstale":"2024-03-28T02:08:54.483056+0000","last_undegraded":"2024-03-28T02:08:54.483056+0000","last_fullsized":"2024-03-28T02:08:54.483056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:47:41.778839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:08.262424+0000","last_change":"2024-03-28T02:08:48.448277+0000","last_active":"2024-03-28T02:09:08.262424+0000","last_peered":"2024-03-28T02:09:08.262424+0000","last_clean":"2024-03-28T02:09:08.262424+0000","last_became_active":"2024-03-28T02:08:48.447910+0000","last_became_peered":"2024-03-28T02:08:48.447910+0000","last_unstale":"2024-03-28T02:09:08.262424+0000","last_undegraded":"2024-03-28T02:09:08.262424+0000","last_fullsized":"2024-03-28T02:09:08.262424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:13:34.241952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873068+0000","last_change":"2024-03-28T02:08:49.258840+0000","last_active":"2024-03-28T02:08:55.873068+0000","last_peered":"2024-03-28T02:08:55.873068+0000","last_clean":"2024-03-28T02:08:55.873068+0000","last_became_active":"2024-03-28T02:08:49.258259+0000","last_became_peered":"2024-03-28T02:08:49.258259+0000","last_unstale":"2024-03-28T02:08:55.873068+0000","last_undegraded":"2024-03-28T02:08:55.873068+0000","last_fullsized":"2024-03-28T02:08:55.873068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:25:31.170413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872856+0000","last_change":"2024-03-28T02:08:53.496476+0000","last_active":"2024-03-28T02:08:55.872856+0000","last_peered":"2024-03-28T02:08:55.872856+0000","last_clean":"2024-03-28T02:08:55.872856+0000","last_became_active":"2024-03-28T02:08:53.496034+0000","last_became_peered":"2024-03-28T02:08:53.496034+0000","last_unstale":"2024-03-28T02:08:55.872856+0000","last_undegraded":"2024-03-28T02:08:55.872856+0000","last_fullsized":"2024-03-28T02:08:55.872856+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:49:00.550877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873186+0000","last_change":"2024-03-28T02:08:48.457960+0000","last_active":"2024-03-28T02:08:55.873186+0000","last_peered":"2024-03-28T02:08:55.873186+0000","last_clean":"2024-03-28T02:08:55.873186+0000","last_became_active":"2024-03-28T02:08:48.455528+0000","last_became_peered":"2024-03-28T02:08:48.455528+0000","last_unstale":"2024-03-28T02:08:55.873186+0000","last_undegraded":"2024-03-28T02:08:55.873186+0000","last_fullsized":"2024-03-28T02:08:55.873186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:40:37.830166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734036+0000","last_change":"2024-03-28T02:08:49.268390+0000","last_active":"2024-03-28T02:08:55.734036+0000","last_peered":"2024-03-28T02:08:55.734036+0000","last_clean":"2024-03-28T02:08:55.734036+0000","last_became_active":"2024-03-28T02:08:49.268025+0000","last_became_peered":"2024-03-28T02:08:49.268025+0000","last_unstale":"2024-03-28T02:08:55.734036+0000","last_undegraded":"2024-03-28T02:08:55.734036+0000","last_fullsized":"2024-03-28T02:08:55.734036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:40:07.752553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873355+0000","last_change":"2024-03-28T02:08:53.492654+0000","last_active":"2024-03-28T02:08:55.873355+0000","last_peered":"2024-03-28T02:08:55.873355+0000","last_clean":"2024-03-28T02:08:55.873355+0000","last_became_active":"2024-03-28T02:08:53.491994+0000","last_became_peered":"2024-03-28T02:08:53.491994+0000","last_unstale":"2024-03-28T02:08:55.873355+0000","last_undegraded":"2024-03-28T02:08:55.873355+0000","last_fullsized":"2024-03-28T02:08:55.873355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:30:55.365074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873730+0000","last_change":"2024-03-28T02:08:48.459667+0000","last_active":"2024-03-28T02:08:55.873730+0000","last_peered":"2024-03-28T02:08:55.873730+0000","last_clean":"2024-03-28T02:08:55.873730+0000","last_became_active":"2024-03-28T02:08:48.459310+0000","last_became_peered":"2024-03-28T02:08:48.459310+0000","last_unstale":"2024-03-28T02:08:55.873730+0000","last_undegraded":"2024-03-28T02:08:55.873730+0000","last_fullsized":"2024-03-28T02:08:55.873730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:51:07.351151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760704+0000","last_change":"2024-03-28T02:08:49.254576+0000","last_active":"2024-03-28T02:08:54.760704+0000","last_peered":"2024-03-28T02:08:54.760704+0000","last_clean":"2024-03-28T02:08:54.760704+0000","last_became_active":"2024-03-28T02:08:49.254213+0000","last_became_peered":"2024-03-28T02:08:49.254213+0000","last_unstale":"2024-03-28T02:08:54.760704+0000","last_undegraded":"2024-03-28T02:08:54.760704+0000","last_fullsized":"2024-03-28T02:08:54.760704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.630801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873534+0000","last_change":"2024-03-28T02:08:53.493624+0000","last_active":"2024-03-28T02:08:55.873534+0000","last_peered":"2024-03-28T02:08:55.873534+0000","last_clean":"2024-03-28T02:08:55.873534+0000","last_became_active":"2024-03-28T02:08:53.493195+0000","last_became_peered":"2024-03-28T02:08:53.493195+0000","last_unstale":"2024-03-28T02:08:55.873534+0000","last_undegraded":"2024-03-28T02:08:55.873534+0000","last_fullsized":"2024-03-28T02:08:55.873534+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:38:25.497464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873375+0000","last_change":"2024-03-28T02:08:48.454008+0000","last_active":"2024-03-28T02:08:55.873375+0000","last_peered":"2024-03-28T02:08:55.873375+0000","last_clean":"2024-03-28T02:08:55.873375+0000","last_became_active":"2024-03-28T02:08:48.450674+0000","last_became_peered":"2024-03-28T02:08:48.450674+0000","last_unstale":"2024-03-28T02:08:55.873375+0000","last_undegraded":"2024-03-28T02:08:55.873375+0000","last_fullsized":"2024-03-28T02:08:55.873375+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:08:12.902348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611109+0000","last_change":"2024-03-28T02:08:49.266552+0000","last_active":"2024-03-28T02:08:55.611109+0000","last_peered":"2024-03-28T02:08:55.611109+0000","last_clean":"2024-03-28T02:08:55.611109+0000","last_became_active":"2024-03-28T02:08:49.265161+0000","last_became_peered":"2024-03-28T02:08:49.265161+0000","last_unstale":"2024-03-28T02:08:55.611109+0000","last_undegraded":"2024-03-28T02:08:55.611109+0000","last_fullsized":"2024-03-28T02:08:55.611109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:22.128908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483449+0000","last_change":"2024-03-28T02:08:53.487871+0000","last_active":"2024-03-28T02:08:54.483449+0000","last_peered":"2024-03-28T02:08:54.483449+0000","last_clean":"2024-03-28T02:08:54.483449+0000","last_became_active":"2024-03-28T02:08:53.487344+0000","last_became_peered":"2024-03-28T02:08:53.487344+0000","last_unstale":"2024-03-28T02:08:54.483449+0000","last_undegraded":"2024-03-28T02:08:54.483449+0000","last_fullsized":"2024-03-28T02:08:54.483449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:34:12.555549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610867+0000","last_change":"2024-03-28T02:08:48.450507+0000","last_active":"2024-03-28T02:08:55.610867+0000","last_peered":"2024-03-28T02:08:55.610867+0000","last_clean":"2024-03-28T02:08:55.610867+0000","last_became_active":"2024-03-28T02:08:48.450065+0000","last_became_peered":"2024-03-28T02:08:48.450065+0000","last_unstale":"2024-03-28T02:08:55.610867+0000","last_undegraded":"2024-03-28T02:08:55.610867+0000","last_fullsized":"2024-03-28T02:08:55.610867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:00:45.393364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873190+0000","last_change":"2024-03-28T02:08:49.256313+0000","last_active":"2024-03-28T02:08:55.873190+0000","last_peered":"2024-03-28T02:08:55.873190+0000","last_clean":"2024-03-28T02:08:55.873190+0000","last_became_active":"2024-03-28T02:08:49.255957+0000","last_became_peered":"2024-03-28T02:08:49.255957+0000","last_unstale":"2024-03-28T02:08:55.873190+0000","last_undegraded":"2024-03-28T02:08:55.873190+0000","last_fullsized":"2024-03-28T02:08:55.873190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:39:27.402983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483351+0000","last_change":"2024-03-28T02:08:53.498554+0000","last_active":"2024-03-28T02:08:54.483351+0000","last_peered":"2024-03-28T02:08:54.483351+0000","last_clean":"2024-03-28T02:08:54.483351+0000","last_became_active":"2024-03-28T02:08:53.498152+0000","last_became_peered":"2024-03-28T02:08:53.498152+0000","last_unstale":"2024-03-28T02:08:54.483351+0000","last_undegraded":"2024-03-28T02:08:54.483351+0000","last_fullsized":"2024-03-28T02:08:54.483351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:41:38.234758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:08.261134+0000","last_change":"2024-03-28T02:08:48.450494+0000","last_active":"2024-03-28T02:09:08.261134+0000","last_peered":"2024-03-28T02:09:08.261134+0000","last_clean":"2024-03-28T02:09:08.261134+0000","last_became_active":"2024-03-28T02:08:48.450158+0000","last_became_peered":"2024-03-28T02:08:48.450158+0000","last_unstale":"2024-03-28T02:09:08.261134+0000","last_undegraded":"2024-03-28T02:09:08.261134+0000","last_fullsized":"2024-03-28T02:09:08.261134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:47:20.021116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611030+0000","last_change":"2024-03-28T02:08:49.265941+0000","last_active":"2024-03-28T02:08:55.611030+0000","last_peered":"2024-03-28T02:08:55.611030+0000","last_clean":"2024-03-28T02:08:55.611030+0000","last_became_active":"2024-03-28T02:08:49.262662+0000","last_became_peered":"2024-03-28T02:08:49.262662+0000","last_unstale":"2024-03-28T02:08:55.611030+0000","last_undegraded":"2024-03-28T02:08:55.611030+0000","last_fullsized":"2024-03-28T02:08:55.611030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:07:23.438391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873535+0000","last_change":"2024-03-28T02:08:53.490046+0000","last_active":"2024-03-28T02:08:55.873535+0000","last_peered":"2024-03-28T02:08:55.873535+0000","last_clean":"2024-03-28T02:08:55.873535+0000","last_became_active":"2024-03-28T02:08:53.489463+0000","last_became_peered":"2024-03-28T02:08:53.489463+0000","last_unstale":"2024-03-28T02:08:55.873535+0000","last_undegraded":"2024-03-28T02:08:55.873535+0000","last_fullsized":"2024-03-28T02:08:55.873535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:25:03.485568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293277+0000","last_change":"2024-03-28T02:08:48.450548+0000","last_active":"2024-03-28T02:08:55.293277+0000","last_peered":"2024-03-28T02:08:55.293277+0000","last_clean":"2024-03-28T02:08:55.293277+0000","last_became_active":"2024-03-28T02:08:48.450068+0000","last_became_peered":"2024-03-28T02:08:48.450068+0000","last_unstale":"2024-03-28T02:08:55.293277+0000","last_undegraded":"2024-03-28T02:08:55.293277+0000","last_fullsized":"2024-03-28T02:08:55.293277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:24:05.231774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872623+0000","last_change":"2024-03-28T02:08:49.262785+0000","last_active":"2024-03-28T02:08:55.872623+0000","last_peered":"2024-03-28T02:08:55.872623+0000","last_clean":"2024-03-28T02:08:55.872623+0000","last_became_active":"2024-03-28T02:08:49.262253+0000","last_became_peered":"2024-03-28T02:08:49.262253+0000","last_unstale":"2024-03-28T02:08:55.872623+0000","last_undegraded":"2024-03-28T02:08:55.872623+0000","last_fullsized":"2024-03-28T02:08:55.872623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:07:37.064693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734109+0000","last_change":"2024-03-28T02:08:53.483738+0000","last_active":"2024-03-28T02:08:55.734109+0000","last_peered":"2024-03-28T02:08:55.734109+0000","last_clean":"2024-03-28T02:08:55.734109+0000","last_became_active":"2024-03-28T02:08:53.483214+0000","last_became_peered":"2024-03-28T02:08:53.483214+0000","last_unstale":"2024-03-28T02:08:55.734109+0000","last_undegraded":"2024-03-28T02:08:55.734109+0000","last_fullsized":"2024-03-28T02:08:55.734109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:06:32.789839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610462+0000","last_change":"2024-03-28T02:08:48.448858+0000","last_active":"2024-03-28T02:08:55.610462+0000","last_peered":"2024-03-28T02:08:55.610462+0000","last_clean":"2024-03-28T02:08:55.610462+0000","last_became_active":"2024-03-28T02:08:48.448435+0000","last_became_peered":"2024-03-28T02:08:48.448435+0000","last_unstale":"2024-03-28T02:08:55.610462+0000","last_undegraded":"2024-03-28T02:08:55.610462+0000","last_fullsized":"2024-03-28T02:08:55.610462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:34:13.035336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482224+0000","last_change":"2024-03-28T02:08:49.261354+0000","last_active":"2024-03-28T02:08:54.482224+0000","last_peered":"2024-03-28T02:08:54.482224+0000","last_clean":"2024-03-28T02:08:54.482224+0000","last_became_active":"2024-03-28T02:08:49.261038+0000","last_became_peered":"2024-03-28T02:08:49.261038+0000","last_unstale":"2024-03-28T02:08:54.482224+0000","last_undegraded":"2024-03-28T02:08:54.482224+0000","last_fullsized":"2024-03-28T02:08:54.482224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:03:15.615484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872750+0000","last_change":"2024-03-28T02:08:53.488573+0000","last_active":"2024-03-28T02:08:55.872750+0000","last_peered":"2024-03-28T02:08:55.872750+0000","last_clean":"2024-03-28T02:08:55.872750+0000","last_became_active":"2024-03-28T02:08:53.488198+0000","last_became_peered":"2024-03-28T02:08:53.488198+0000","last_unstale":"2024-03-28T02:08:55.872750+0000","last_undegraded":"2024-03-28T02:08:55.872750+0000","last_fullsized":"2024-03-28T02:08:55.872750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:43:29.158223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:07.533049+0000","last_change":"2024-03-28T02:08:48.445941+0000","last_active":"2024-03-28T02:09:07.533049+0000","last_peered":"2024-03-28T02:09:07.533049+0000","last_clean":"2024-03-28T02:09:07.533049+0000","last_became_active":"2024-03-28T02:08:48.445599+0000","last_became_peered":"2024-03-28T02:08:48.445599+0000","last_unstale":"2024-03-28T02:09:07.533049+0000","last_undegraded":"2024-03-28T02:09:07.533049+0000","last_fullsized":"2024-03-28T02:09:07.533049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:28:53.524708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760857+0000","last_change":"2024-03-28T02:08:49.258592+0000","last_active":"2024-03-28T02:08:54.760857+0000","last_peered":"2024-03-28T02:08:54.760857+0000","last_clean":"2024-03-28T02:08:54.760857+0000","last_became_active":"2024-03-28T02:08:49.256527+0000","last_became_peered":"2024-03-28T02:08:49.256527+0000","last_unstale":"2024-03-28T02:08:54.760857+0000","last_undegraded":"2024-03-28T02:08:54.760857+0000","last_fullsized":"2024-03-28T02:08:54.760857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:54:41.645210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482255+0000","last_change":"2024-03-28T02:08:53.494677+0000","last_active":"2024-03-28T02:08:54.482255+0000","last_peered":"2024-03-28T02:08:54.482255+0000","last_clean":"2024-03-28T02:08:54.482255+0000","last_became_active":"2024-03-28T02:08:53.493076+0000","last_became_peered":"2024-03-28T02:08:53.493076+0000","last_unstale":"2024-03-28T02:08:54.482255+0000","last_undegraded":"2024-03-28T02:08:54.482255+0000","last_fullsized":"2024-03-28T02:08:54.482255+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:10:44.817473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873285+0000","last_change":"2024-03-28T02:08:48.454321+0000","last_active":"2024-03-28T02:08:55.873285+0000","last_peered":"2024-03-28T02:08:55.873285+0000","last_clean":"2024-03-28T02:08:55.873285+0000","last_became_active":"2024-03-28T02:08:48.453907+0000","last_became_peered":"2024-03-28T02:08:48.453907+0000","last_unstale":"2024-03-28T02:08:55.873285+0000","last_undegraded":"2024-03-28T02:08:55.873285+0000","last_fullsized":"2024-03-28T02:08:55.873285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:45:18.347073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734156+0000","last_change":"2024-03-28T02:08:49.262196+0000","last_active":"2024-03-28T02:08:55.734156+0000","last_peered":"2024-03-28T02:08:55.734156+0000","last_clean":"2024-03-28T02:08:55.734156+0000","last_became_active":"2024-03-28T02:08:49.261190+0000","last_became_peered":"2024-03-28T02:08:49.261190+0000","last_unstale":"2024-03-28T02:08:55.734156+0000","last_undegraded":"2024-03-28T02:08:55.734156+0000","last_fullsized":"2024-03-28T02:08:55.734156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:37:15.939535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873615+0000","last_change":"2024-03-28T02:08:53.492207+0000","last_active":"2024-03-28T02:08:55.873615+0000","last_peered":"2024-03-28T02:08:55.873615+0000","last_clean":"2024-03-28T02:08:55.873615+0000","last_became_active":"2024-03-28T02:08:53.491831+0000","last_became_peered":"2024-03-28T02:08:53.491831+0000","last_unstale":"2024-03-28T02:08:55.873615+0000","last_undegraded":"2024-03-28T02:08:55.873615+0000","last_fullsized":"2024-03-28T02:08:55.873615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:00.163397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610720+0000","last_change":"2024-03-28T02:08:48.450908+0000","last_active":"2024-03-28T02:08:55.610720+0000","last_peered":"2024-03-28T02:08:55.610720+0000","last_clean":"2024-03-28T02:08:55.610720+0000","last_became_active":"2024-03-28T02:08:48.450382+0000","last_became_peered":"2024-03-28T02:08:48.450382+0000","last_unstale":"2024-03-28T02:08:55.610720+0000","last_undegraded":"2024-03-28T02:08:55.610720+0000","last_fullsized":"2024-03-28T02:08:55.610720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:26:25.851416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760801+0000","last_change":"2024-03-28T02:08:49.268904+0000","last_active":"2024-03-28T02:08:54.760801+0000","last_peered":"2024-03-28T02:08:54.760801+0000","last_clean":"2024-03-28T02:08:54.760801+0000","last_became_active":"2024-03-28T02:08:49.268595+0000","last_became_peered":"2024-03-28T02:08:49.268595+0000","last_unstale":"2024-03-28T02:08:54.760801+0000","last_undegraded":"2024-03-28T02:08:54.760801+0000","last_fullsized":"2024-03-28T02:08:54.760801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:12:25.903861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482278+0000","last_change":"2024-03-28T02:08:53.489115+0000","last_active":"2024-03-28T02:08:54.482278+0000","last_peered":"2024-03-28T02:08:54.482278+0000","last_clean":"2024-03-28T02:08:54.482278+0000","last_became_active":"2024-03-28T02:08:53.488716+0000","last_became_peered":"2024-03-28T02:08:53.488716+0000","last_unstale":"2024-03-28T02:08:54.482278+0000","last_undegraded":"2024-03-28T02:08:54.482278+0000","last_fullsized":"2024-03-28T02:08:54.482278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:00:42.920205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:08.250333+0000","last_change":"2024-03-28T02:08:48.453921+0000","last_active":"2024-03-28T02:09:08.250333+0000","last_peered":"2024-03-28T02:09:08.250333+0000","last_clean":"2024-03-28T02:09:08.250333+0000","last_became_active":"2024-03-28T02:08:48.453301+0000","last_became_peered":"2024-03-28T02:08:48.453301+0000","last_unstale":"2024-03-28T02:09:08.250333+0000","last_undegraded":"2024-03-28T02:09:08.250333+0000","last_fullsized":"2024-03-28T02:09:08.250333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:57:04.543444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873383+0000","last_change":"2024-03-28T02:08:49.259942+0000","last_active":"2024-03-28T02:08:55.873383+0000","last_peered":"2024-03-28T02:08:55.873383+0000","last_clean":"2024-03-28T02:08:55.873383+0000","last_became_active":"2024-03-28T02:08:49.259602+0000","last_became_peered":"2024-03-28T02:08:49.259602+0000","last_unstale":"2024-03-28T02:08:55.873383+0000","last_undegraded":"2024-03-28T02:08:55.873383+0000","last_fullsized":"2024-03-28T02:08:55.873383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:38:05.897720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611464+0000","last_change":"2024-03-28T02:08:53.495675+0000","last_active":"2024-03-28T02:08:55.611464+0000","last_peered":"2024-03-28T02:08:55.611464+0000","last_clean":"2024-03-28T02:08:55.611464+0000","last_became_active":"2024-03-28T02:08:53.495233+0000","last_became_peered":"2024-03-28T02:08:53.495233+0000","last_unstale":"2024-03-28T02:08:55.611464+0000","last_undegraded":"2024-03-28T02:08:55.611464+0000","last_fullsized":"2024-03-28T02:08:55.611464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:43.805151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483024+0000","last_change":"2024-03-28T02:08:48.452335+0000","last_active":"2024-03-28T02:08:54.483024+0000","last_peered":"2024-03-28T02:08:54.483024+0000","last_clean":"2024-03-28T02:08:54.483024+0000","last_became_active":"2024-03-28T02:08:48.449966+0000","last_became_peered":"2024-03-28T02:08:48.449966+0000","last_unstale":"2024-03-28T02:08:54.483024+0000","last_undegraded":"2024-03-28T02:08:54.483024+0000","last_fullsized":"2024-03-28T02:08:54.483024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:20:09.757746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293426+0000","last_change":"2024-03-28T02:08:49.259179+0000","last_active":"2024-03-28T02:08:55.293426+0000","last_peered":"2024-03-28T02:08:55.293426+0000","last_clean":"2024-03-28T02:08:55.293426+0000","last_became_active":"2024-03-28T02:08:49.258538+0000","last_became_peered":"2024-03-28T02:08:49.258538+0000","last_unstale":"2024-03-28T02:08:55.293426+0000","last_undegraded":"2024-03-28T02:08:55.293426+0000","last_fullsized":"2024-03-28T02:08:55.293426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:12:20.008528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873479+0000","last_change":"2024-03-28T02:08:53.499523+0000","last_active":"2024-03-28T02:08:55.873479+0000","last_peered":"2024-03-28T02:08:55.873479+0000","last_clean":"2024-03-28T02:08:55.873479+0000","last_became_active":"2024-03-28T02:08:53.499112+0000","last_became_peered":"2024-03-28T02:08:53.499112+0000","last_unstale":"2024-03-28T02:08:55.873479+0000","last_undegraded":"2024-03-28T02:08:55.873479+0000","last_fullsized":"2024-03-28T02:08:55.873479+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:31:12.008640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610781+0000","last_change":"2024-03-28T02:08:48.452952+0000","last_active":"2024-03-28T02:08:55.610781+0000","last_peered":"2024-03-28T02:08:55.610781+0000","last_clean":"2024-03-28T02:08:55.610781+0000","last_became_active":"2024-03-28T02:08:48.452586+0000","last_became_peered":"2024-03-28T02:08:48.452586+0000","last_unstale":"2024-03-28T02:08:55.610781+0000","last_undegraded":"2024-03-28T02:08:55.610781+0000","last_fullsized":"2024-03-28T02:08:55.610781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:44.749902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873723+0000","last_change":"2024-03-28T02:08:49.256746+0000","last_active":"2024-03-28T02:08:55.873723+0000","last_peered":"2024-03-28T02:08:55.873723+0000","last_clean":"2024-03-28T02:08:55.873723+0000","last_became_active":"2024-03-28T02:08:49.256398+0000","last_became_peered":"2024-03-28T02:08:49.256398+0000","last_unstale":"2024-03-28T02:08:55.873723+0000","last_undegraded":"2024-03-28T02:08:55.873723+0000","last_fullsized":"2024-03-28T02:08:55.873723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:10:52.021218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482830+0000","last_change":"2024-03-28T02:08:53.494257+0000","last_active":"2024-03-28T02:08:54.482830+0000","last_peered":"2024-03-28T02:08:54.482830+0000","last_clean":"2024-03-28T02:08:54.482830+0000","last_became_active":"2024-03-28T02:08:53.493889+0000","last_became_peered":"2024-03-28T02:08:53.493889+0000","last_unstale":"2024-03-28T02:08:54.482830+0000","last_undegraded":"2024-03-28T02:08:54.482830+0000","last_fullsized":"2024-03-28T02:08:54.482830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:49.292686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872868+0000","last_change":"2024-03-28T02:08:48.447361+0000","last_active":"2024-03-28T02:08:55.872868+0000","last_peered":"2024-03-28T02:08:55.872868+0000","last_clean":"2024-03-28T02:08:55.872868+0000","last_became_active":"2024-03-28T02:08:48.446943+0000","last_became_peered":"2024-03-28T02:08:48.446943+0000","last_unstale":"2024-03-28T02:08:55.872868+0000","last_undegraded":"2024-03-28T02:08:55.872868+0000","last_fullsized":"2024-03-28T02:08:55.872868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:19:46.552447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872881+0000","last_change":"2024-03-28T02:08:49.255253+0000","last_active":"2024-03-28T02:08:55.872881+0000","last_peered":"2024-03-28T02:08:55.872881+0000","last_clean":"2024-03-28T02:08:55.872881+0000","last_became_active":"2024-03-28T02:08:49.254804+0000","last_became_peered":"2024-03-28T02:08:49.254804+0000","last_unstale":"2024-03-28T02:08:55.872881+0000","last_undegraded":"2024-03-28T02:08:55.872881+0000","last_fullsized":"2024-03-28T02:08:55.872881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:53:31.927830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293778+0000","last_change":"2024-03-28T02:08:53.497820+0000","last_active":"2024-03-28T02:08:55.293778+0000","last_peered":"2024-03-28T02:08:55.293778+0000","last_clean":"2024-03-28T02:08:55.293778+0000","last_became_active":"2024-03-28T02:08:53.497441+0000","last_became_peered":"2024-03-28T02:08:53.497441+0000","last_unstale":"2024-03-28T02:08:55.293778+0000","last_undegraded":"2024-03-28T02:08:55.293778+0000","last_fullsized":"2024-03-28T02:08:55.293778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:12:02.993121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873023+0000","last_change":"2024-03-28T02:08:48.445907+0000","last_active":"2024-03-28T02:08:55.873023+0000","last_peered":"2024-03-28T02:08:55.873023+0000","last_clean":"2024-03-28T02:08:55.873023+0000","last_became_active":"2024-03-28T02:08:48.444278+0000","last_became_peered":"2024-03-28T02:08:48.444278+0000","last_unstale":"2024-03-28T02:08:55.873023+0000","last_undegraded":"2024-03-28T02:08:55.873023+0000","last_fullsized":"2024-03-28T02:08:55.873023+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:23:30.060259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610584+0000","last_change":"2024-03-28T02:08:49.262862+0000","last_active":"2024-03-28T02:08:55.610584+0000","last_peered":"2024-03-28T02:08:55.610584+0000","last_clean":"2024-03-28T02:08:55.610584+0000","last_became_active":"2024-03-28T02:08:49.262451+0000","last_became_peered":"2024-03-28T02:08:49.262451+0000","last_unstale":"2024-03-28T02:08:55.610584+0000","last_undegraded":"2024-03-28T02:08:55.610584+0000","last_fullsized":"2024-03-28T02:08:55.610584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:12:51.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610891+0000","last_change":"2024-03-28T02:08:53.490855+0000","last_active":"2024-03-28T02:08:55.610891+0000","last_peered":"2024-03-28T02:08:55.610891+0000","last_clean":"2024-03-28T02:08:55.610891+0000","last_became_active":"2024-03-28T02:08:53.490415+0000","last_became_peered":"2024-03-28T02:08:53.490415+0000","last_unstale":"2024-03-28T02:08:55.610891+0000","last_undegraded":"2024-03-28T02:08:55.610891+0000","last_fullsized":"2024-03-28T02:08:55.610891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:56:37.426398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873783+0000","last_change":"2024-03-28T02:08:48.448909+0000","last_active":"2024-03-28T02:08:55.873783+0000","last_peered":"2024-03-28T02:08:55.873783+0000","last_clean":"2024-03-28T02:08:55.873783+0000","last_became_active":"2024-03-28T02:08:48.448548+0000","last_became_peered":"2024-03-28T02:08:48.448548+0000","last_unstale":"2024-03-28T02:08:55.873783+0000","last_undegraded":"2024-03-28T02:08:55.873783+0000","last_fullsized":"2024-03-28T02:08:55.873783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:51:06.206266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873065+0000","last_change":"2024-03-28T02:08:49.266920+0000","last_active":"2024-03-28T02:08:55.873065+0000","last_peered":"2024-03-28T02:08:55.873065+0000","last_clean":"2024-03-28T02:08:55.873065+0000","last_became_active":"2024-03-28T02:08:49.266600+0000","last_became_peered":"2024-03-28T02:08:49.266600+0000","last_unstale":"2024-03-28T02:08:55.873065+0000","last_undegraded":"2024-03-28T02:08:55.873065+0000","last_fullsized":"2024-03-28T02:08:55.873065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:57.735460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873183+0000","last_change":"2024-03-28T02:08:53.631636+0000","last_active":"2024-03-28T02:08:55.873183+0000","last_peered":"2024-03-28T02:08:55.873183+0000","last_clean":"2024-03-28T02:08:55.873183+0000","last_became_active":"2024-03-28T02:08:53.631183+0000","last_became_peered":"2024-03-28T02:08:53.631183+0000","last_unstale":"2024-03-28T02:08:55.873183+0000","last_undegraded":"2024-03-28T02:08:55.873183+0000","last_fullsized":"2024-03-28T02:08:55.873183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:22:09.212726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482576+0000","last_change":"2024-03-28T02:08:48.440789+0000","last_active":"2024-03-28T02:08:54.482576+0000","last_peered":"2024-03-28T02:08:54.482576+0000","last_clean":"2024-03-28T02:08:54.482576+0000","last_became_active":"2024-03-28T02:08:48.440403+0000","last_became_peered":"2024-03-28T02:08:48.440403+0000","last_unstale":"2024-03-28T02:08:54.482576+0000","last_undegraded":"2024-03-28T02:08:54.482576+0000","last_fullsized":"2024-03-28T02:08:54.482576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:37.227279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482918+0000","last_change":"2024-03-28T02:08:49.255340+0000","last_active":"2024-03-28T02:08:54.482918+0000","last_peered":"2024-03-28T02:08:54.482918+0000","last_clean":"2024-03-28T02:08:54.482918+0000","last_became_active":"2024-03-28T02:08:49.255027+0000","last_became_peered":"2024-03-28T02:08:49.255027+0000","last_unstale":"2024-03-28T02:08:54.482918+0000","last_undegraded":"2024-03-28T02:08:54.482918+0000","last_fullsized":"2024-03-28T02:08:54.482918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:54:03.454022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293642+0000","last_change":"2024-03-28T02:08:53.499721+0000","last_active":"2024-03-28T02:08:55.293642+0000","last_peered":"2024-03-28T02:08:55.293642+0000","last_clean":"2024-03-28T02:08:55.293642+0000","last_became_active":"2024-03-28T02:08:53.499117+0000","last_became_peered":"2024-03-28T02:08:53.499117+0000","last_unstale":"2024-03-28T02:08:55.293642+0000","last_undegraded":"2024-03-28T02:08:55.293642+0000","last_fullsized":"2024-03-28T02:08:55.293642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:07:33.561249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873088+0000","last_change":"2024-03-28T02:08:48.453592+0000","last_active":"2024-03-28T02:08:55.873088+0000","last_peered":"2024-03-28T02:08:55.873088+0000","last_clean":"2024-03-28T02:08:55.873088+0000","last_became_active":"2024-03-28T02:08:48.453169+0000","last_became_peered":"2024-03-28T02:08:48.453169+0000","last_unstale":"2024-03-28T02:08:55.873088+0000","last_undegraded":"2024-03-28T02:08:55.873088+0000","last_fullsized":"2024-03-28T02:08:55.873088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:33.163170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482700+0000","last_change":"2024-03-28T02:08:49.263143+0000","last_active":"2024-03-28T02:08:54.482700+0000","last_peered":"2024-03-28T02:08:54.482700+0000","last_clean":"2024-03-28T02:08:54.482700+0000","last_became_active":"2024-03-28T02:08:49.262829+0000","last_became_peered":"2024-03-28T02:08:49.262829+0000","last_unstale":"2024-03-28T02:08:54.482700+0000","last_undegraded":"2024-03-28T02:08:54.482700+0000","last_fullsized":"2024-03-28T02:08:54.482700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:01:07.809685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293785+0000","last_change":"2024-03-28T02:08:53.631595+0000","last_active":"2024-03-28T02:08:55.293785+0000","last_peered":"2024-03-28T02:08:55.293785+0000","last_clean":"2024-03-28T02:08:55.293785+0000","last_became_active":"2024-03-28T02:08:53.630863+0000","last_became_peered":"2024-03-28T02:08:53.630863+0000","last_unstale":"2024-03-28T02:08:55.293785+0000","last_undegraded":"2024-03-28T02:08:55.293785+0000","last_fullsized":"2024-03-28T02:08:55.293785+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:38:44.712728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610770+0000","last_change":"2024-03-28T02:08:49.263523+0000","last_active":"2024-03-28T02:08:55.610770+0000","last_peered":"2024-03-28T02:08:55.610770+0000","last_clean":"2024-03-28T02:08:55.610770+0000","last_became_active":"2024-03-28T02:08:49.263079+0000","last_became_peered":"2024-03-28T02:08:49.263079+0000","last_unstale":"2024-03-28T02:08:55.610770+0000","last_undegraded":"2024-03-28T02:08:55.610770+0000","last_fullsized":"2024-03-28T02:08:55.610770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:15:37.516032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293168+0000","last_change":"2024-03-28T02:08:48.455723+0000","last_active":"2024-03-28T02:08:55.293168+0000","last_peered":"2024-03-28T02:08:55.293168+0000","last_clean":"2024-03-28T02:08:55.293168+0000","last_became_active":"2024-03-28T02:08:48.455340+0000","last_became_peered":"2024-03-28T02:08:48.455340+0000","last_unstale":"2024-03-28T02:08:55.293168+0000","last_undegraded":"2024-03-28T02:08:55.293168+0000","last_fullsized":"2024-03-28T02:08:55.293168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:15:36.376221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873515+0000","last_change":"2024-03-28T02:08:53.486656+0000","last_active":"2024-03-28T02:08:55.873515+0000","last_peered":"2024-03-28T02:08:55.873515+0000","last_clean":"2024-03-28T02:08:55.873515+0000","last_became_active":"2024-03-28T02:08:53.486274+0000","last_became_peered":"2024-03-28T02:08:53.486274+0000","last_unstale":"2024-03-28T02:08:55.873515+0000","last_undegraded":"2024-03-28T02:08:55.873515+0000","last_fullsized":"2024-03-28T02:08:55.873515+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:40:40.742751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872651+0000","last_change":"2024-03-28T02:08:49.260380+0000","last_active":"2024-03-28T02:08:55.872651+0000","last_peered":"2024-03-28T02:08:55.872651+0000","last_clean":"2024-03-28T02:08:55.872651+0000","last_became_active":"2024-03-28T02:08:49.260012+0000","last_became_peered":"2024-03-28T02:08:49.260012+0000","last_unstale":"2024-03-28T02:08:55.872651+0000","last_undegraded":"2024-03-28T02:08:55.872651+0000","last_fullsized":"2024-03-28T02:08:55.872651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:46:31.233692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873406+0000","last_change":"2024-03-28T02:08:48.447659+0000","last_active":"2024-03-28T02:08:55.873406+0000","last_peered":"2024-03-28T02:08:55.873406+0000","last_clean":"2024-03-28T02:08:55.873406+0000","last_became_active":"2024-03-28T02:08:48.447263+0000","last_became_peered":"2024-03-28T02:08:48.447263+0000","last_unstale":"2024-03-28T02:08:55.873406+0000","last_undegraded":"2024-03-28T02:08:55.873406+0000","last_fullsized":"2024-03-28T02:08:55.873406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:16:32.453528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872752+0000","last_change":"2024-03-28T02:07:33.021769+0000","last_active":"2024-03-28T02:08:55.872752+0000","last_peered":"2024-03-28T02:08:55.872752+0000","last_clean":"2024-03-28T02:08:55.872752+0000","last_became_active":"2024-03-28T02:07:33.021338+0000","last_became_peered":"2024-03-28T02:07:33.021338+0000","last_unstale":"2024-03-28T02:08:55.872752+0000","last_undegraded":"2024-03-28T02:08:55.872752+0000","last_fullsized":"2024-03-28T02:08:55.872752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:04:07.152098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:04:07.152098+0000","last_clean_scrub_stamp":"2024-03-28T02:04:07.152098+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:01:39.981299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611268+0000","last_change":"2024-03-28T02:08:53.631687+0000","last_active":"2024-03-28T02:08:55.611268+0000","last_peered":"2024-03-28T02:08:55.611268+0000","last_clean":"2024-03-28T02:08:55.611268+0000","last_became_active":"2024-03-28T02:08:53.631291+0000","last_became_peered":"2024-03-28T02:08:53.631291+0000","last_unstale":"2024-03-28T02:08:55.611268+0000","last_undegraded":"2024-03-28T02:08:55.611268+0000","last_fullsized":"2024-03-28T02:08:55.611268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:18:51.563667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293163+0000","last_change":"2024-03-28T02:08:49.255310+0000","last_active":"2024-03-28T02:08:55.293163+0000","last_peered":"2024-03-28T02:08:55.293163+0000","last_clean":"2024-03-28T02:08:55.293163+0000","last_became_active":"2024-03-28T02:08:49.254923+0000","last_became_peered":"2024-03-28T02:08:49.254923+0000","last_unstale":"2024-03-28T02:08:55.293163+0000","last_undegraded":"2024-03-28T02:08:55.293163+0000","last_fullsized":"2024-03-28T02:08:55.293163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:34.660601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873419+0000","last_change":"2024-03-28T02:08:48.457646+0000","last_active":"2024-03-28T02:08:55.873419+0000","last_peered":"2024-03-28T02:08:55.873419+0000","last_clean":"2024-03-28T02:08:55.873419+0000","last_became_active":"2024-03-28T02:08:48.457344+0000","last_became_peered":"2024-03-28T02:08:48.457344+0000","last_unstale":"2024-03-28T02:08:55.873419+0000","last_undegraded":"2024-03-28T02:08:55.873419+0000","last_fullsized":"2024-03-28T02:08:55.873419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:25:19.077816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873072+0000","last_change":"2024-03-28T02:08:53.486320+0000","last_active":"2024-03-28T02:08:55.873072+0000","last_peered":"2024-03-28T02:08:55.873072+0000","last_clean":"2024-03-28T02:08:55.873072+0000","last_became_active":"2024-03-28T02:08:53.485771+0000","last_became_peered":"2024-03-28T02:08:53.485771+0000","last_unstale":"2024-03-28T02:08:55.873072+0000","last_undegraded":"2024-03-28T02:08:55.873072+0000","last_fullsized":"2024-03-28T02:08:55.873072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:48.452990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873555+0000","last_change":"2024-03-28T02:08:48.458854+0000","last_active":"2024-03-28T02:08:55.873555+0000","last_peered":"2024-03-28T02:08:55.873555+0000","last_clean":"2024-03-28T02:08:55.873555+0000","last_became_active":"2024-03-28T02:08:48.456667+0000","last_became_peered":"2024-03-28T02:08:48.456667+0000","last_unstale":"2024-03-28T02:08:55.873555+0000","last_undegraded":"2024-03-28T02:08:55.873555+0000","last_fullsized":"2024-03-28T02:08:55.873555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:37:54.113797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293659+0000","last_change":"2024-03-28T02:08:49.255300+0000","last_active":"2024-03-28T02:08:55.293659+0000","last_peered":"2024-03-28T02:08:55.293659+0000","last_clean":"2024-03-28T02:08:55.293659+0000","last_became_active":"2024-03-28T02:08:49.254955+0000","last_became_peered":"2024-03-28T02:08:49.254955+0000","last_unstale":"2024-03-28T02:08:55.293659+0000","last_undegraded":"2024-03-28T02:08:55.293659+0000","last_fullsized":"2024-03-28T02:08:55.293659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:29.093770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872623+0000","last_change":"2024-03-28T02:08:53.631966+0000","last_active":"2024-03-28T02:08:55.872623+0000","last_peered":"2024-03-28T02:08:55.872623+0000","last_clean":"2024-03-28T02:08:55.872623+0000","last_became_active":"2024-03-28T02:08:53.631457+0000","last_became_peered":"2024-03-28T02:08:53.631457+0000","last_unstale":"2024-03-28T02:08:55.872623+0000","last_undegraded":"2024-03-28T02:08:55.872623+0000","last_fullsized":"2024-03-28T02:08:55.872623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:36:46.623567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:09:08.258986+0000","last_change":"2024-03-28T02:08:48.452991+0000","last_active":"2024-03-28T02:09:08.258986+0000","last_peered":"2024-03-28T02:09:08.258986+0000","last_clean":"2024-03-28T02:09:08.258986+0000","last_became_active":"2024-03-28T02:08:48.452248+0000","last_became_peered":"2024-03-28T02:08:48.452248+0000","last_unstale":"2024-03-28T02:09:08.258986+0000","last_undegraded":"2024-03-28T02:09:08.258986+0000","last_fullsized":"2024-03-28T02:09:08.258986+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:13:40.100006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873065+0000","last_change":"2024-03-28T02:08:49.259575+0000","last_active":"2024-03-28T02:08:55.873065+0000","last_peered":"2024-03-28T02:08:55.873065+0000","last_clean":"2024-03-28T02:08:55.873065+0000","last_became_active":"2024-03-28T02:08:49.259255+0000","last_became_peered":"2024-03-28T02:08:49.259255+0000","last_unstale":"2024-03-28T02:08:55.873065+0000","last_undegraded":"2024-03-28T02:08:55.873065+0000","last_fullsized":"2024-03-28T02:08:55.873065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:47:17.741691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482818+0000","last_change":"2024-03-28T02:08:53.488304+0000","last_active":"2024-03-28T02:08:54.482818+0000","last_peered":"2024-03-28T02:08:54.482818+0000","last_clean":"2024-03-28T02:08:54.482818+0000","last_became_active":"2024-03-28T02:08:53.487783+0000","last_became_peered":"2024-03-28T02:08:53.487783+0000","last_unstale":"2024-03-28T02:08:54.482818+0000","last_undegraded":"2024-03-28T02:08:54.482818+0000","last_fullsized":"2024-03-28T02:08:54.482818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:54:50.494685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873080+0000","last_change":"2024-03-28T02:08:48.450795+0000","last_active":"2024-03-28T02:08:55.873080+0000","last_peered":"2024-03-28T02:08:55.873080+0000","last_clean":"2024-03-28T02:08:55.873080+0000","last_became_active":"2024-03-28T02:08:48.450449+0000","last_became_peered":"2024-03-28T02:08:48.450449+0000","last_unstale":"2024-03-28T02:08:55.873080+0000","last_undegraded":"2024-03-28T02:08:55.873080+0000","last_fullsized":"2024-03-28T02:08:55.873080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:44.576188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734045+0000","last_change":"2024-03-28T02:08:49.264220+0000","last_active":"2024-03-28T02:08:55.734045+0000","last_peered":"2024-03-28T02:08:55.734045+0000","last_clean":"2024-03-28T02:08:55.734045+0000","last_became_active":"2024-03-28T02:08:49.263757+0000","last_became_peered":"2024-03-28T02:08:49.263757+0000","last_unstale":"2024-03-28T02:08:55.734045+0000","last_undegraded":"2024-03-28T02:08:55.734045+0000","last_fullsized":"2024-03-28T02:08:55.734045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:20:44.000256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873280+0000","last_change":"2024-03-28T02:08:53.631904+0000","last_active":"2024-03-28T02:08:55.873280+0000","last_peered":"2024-03-28T02:08:55.873280+0000","last_clean":"2024-03-28T02:08:55.873280+0000","last_became_active":"2024-03-28T02:08:53.631263+0000","last_became_peered":"2024-03-28T02:08:53.631263+0000","last_unstale":"2024-03-28T02:08:55.873280+0000","last_undegraded":"2024-03-28T02:08:55.873280+0000","last_fullsized":"2024-03-28T02:08:55.873280+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:30:53.511632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760357+0000","last_change":"2024-03-28T02:08:48.450272+0000","last_active":"2024-03-28T02:08:54.760357+0000","last_peered":"2024-03-28T02:08:54.760357+0000","last_clean":"2024-03-28T02:08:54.760357+0000","last_became_active":"2024-03-28T02:08:48.449905+0000","last_became_peered":"2024-03-28T02:08:48.449905+0000","last_unstale":"2024-03-28T02:08:54.760357+0000","last_undegraded":"2024-03-28T02:08:54.760357+0000","last_fullsized":"2024-03-28T02:08:54.760357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:56:49.493569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873168+0000","last_change":"2024-03-28T02:08:49.256816+0000","last_active":"2024-03-28T02:08:55.873168+0000","last_peered":"2024-03-28T02:08:55.873168+0000","last_clean":"2024-03-28T02:08:55.873168+0000","last_became_active":"2024-03-28T02:08:49.256369+0000","last_became_peered":"2024-03-28T02:08:49.256369+0000","last_unstale":"2024-03-28T02:08:55.873168+0000","last_undegraded":"2024-03-28T02:08:55.873168+0000","last_fullsized":"2024-03-28T02:08:55.873168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:39.714619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482907+0000","last_change":"2024-03-28T02:08:53.496231+0000","last_active":"2024-03-28T02:08:54.482907+0000","last_peered":"2024-03-28T02:08:54.482907+0000","last_clean":"2024-03-28T02:08:54.482907+0000","last_became_active":"2024-03-28T02:08:53.494179+0000","last_became_peered":"2024-03-28T02:08:53.494179+0000","last_unstale":"2024-03-28T02:08:54.482907+0000","last_undegraded":"2024-03-28T02:08:54.482907+0000","last_fullsized":"2024-03-28T02:08:54.482907+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:17:17.231150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872575+0000","last_change":"2024-03-28T02:08:48.454998+0000","last_active":"2024-03-28T02:08:55.872575+0000","last_peered":"2024-03-28T02:08:55.872575+0000","last_clean":"2024-03-28T02:08:55.872575+0000","last_became_active":"2024-03-28T02:08:48.454674+0000","last_became_peered":"2024-03-28T02:08:48.454674+0000","last_unstale":"2024-03-28T02:08:55.872575+0000","last_undegraded":"2024-03-28T02:08:55.872575+0000","last_fullsized":"2024-03-28T02:08:55.872575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:01:23.432625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872754+0000","last_change":"2024-03-28T02:08:49.263714+0000","last_active":"2024-03-28T02:08:55.872754+0000","last_peered":"2024-03-28T02:08:55.872754+0000","last_clean":"2024-03-28T02:08:55.872754+0000","last_became_active":"2024-03-28T02:08:49.263335+0000","last_became_peered":"2024-03-28T02:08:49.263335+0000","last_unstale":"2024-03-28T02:08:55.872754+0000","last_undegraded":"2024-03-28T02:08:55.872754+0000","last_fullsized":"2024-03-28T02:08:55.872754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:28:13.153319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873827+0000","last_change":"2024-03-28T02:08:53.496751+0000","last_active":"2024-03-28T02:08:55.873827+0000","last_peered":"2024-03-28T02:08:55.873827+0000","last_clean":"2024-03-28T02:08:55.873827+0000","last_became_active":"2024-03-28T02:08:53.496393+0000","last_became_peered":"2024-03-28T02:08:53.496393+0000","last_unstale":"2024-03-28T02:08:55.873827+0000","last_undegraded":"2024-03-28T02:08:55.873827+0000","last_fullsized":"2024-03-28T02:08:55.873827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:44:13.268278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482840+0000","last_change":"2024-03-28T02:08:48.456026+0000","last_active":"2024-03-28T02:08:54.482840+0000","last_peered":"2024-03-28T02:08:54.482840+0000","last_clean":"2024-03-28T02:08:54.482840+0000","last_became_active":"2024-03-28T02:08:48.455264+0000","last_became_peered":"2024-03-28T02:08:48.455264+0000","last_unstale":"2024-03-28T02:08:54.482840+0000","last_undegraded":"2024-03-28T02:08:54.482840+0000","last_fullsized":"2024-03-28T02:08:54.482840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:21:51.682413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872934+0000","last_change":"2024-03-28T02:08:49.260416+0000","last_active":"2024-03-28T02:08:55.872934+0000","last_peered":"2024-03-28T02:08:55.872934+0000","last_clean":"2024-03-28T02:08:55.872934+0000","last_became_active":"2024-03-28T02:08:49.260071+0000","last_became_peered":"2024-03-28T02:08:49.260071+0000","last_unstale":"2024-03-28T02:08:55.872934+0000","last_undegraded":"2024-03-28T02:08:55.872934+0000","last_fullsized":"2024-03-28T02:08:55.872934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:34:56.414305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873021+0000","last_change":"2024-03-28T02:08:53.631633+0000","last_active":"2024-03-28T02:08:55.873021+0000","last_peered":"2024-03-28T02:08:55.873021+0000","last_clean":"2024-03-28T02:08:55.873021+0000","last_became_active":"2024-03-28T02:08:53.631158+0000","last_became_peered":"2024-03-28T02:08:53.631158+0000","last_unstale":"2024-03-28T02:08:55.873021+0000","last_undegraded":"2024-03-28T02:08:55.873021+0000","last_fullsized":"2024-03-28T02:08:55.873021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:37:43.749860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873453+0000","last_change":"2024-03-28T02:08:48.454468+0000","last_active":"2024-03-28T02:08:55.873453+0000","last_peered":"2024-03-28T02:08:55.873453+0000","last_clean":"2024-03-28T02:08:55.873453+0000","last_became_active":"2024-03-28T02:08:48.454086+0000","last_became_peered":"2024-03-28T02:08:48.454086+0000","last_unstale":"2024-03-28T02:08:55.873453+0000","last_undegraded":"2024-03-28T02:08:55.873453+0000","last_fullsized":"2024-03-28T02:08:55.873453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:03.178254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873118+0000","last_change":"2024-03-28T02:08:49.268529+0000","last_active":"2024-03-28T02:08:55.873118+0000","last_peered":"2024-03-28T02:08:55.873118+0000","last_clean":"2024-03-28T02:08:55.873118+0000","last_became_active":"2024-03-28T02:08:49.267869+0000","last_became_peered":"2024-03-28T02:08:49.267869+0000","last_unstale":"2024-03-28T02:08:55.873118+0000","last_undegraded":"2024-03-28T02:08:55.873118+0000","last_fullsized":"2024-03-28T02:08:55.873118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:52:34.633616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760699+0000","last_change":"2024-03-28T02:08:53.495894+0000","last_active":"2024-03-28T02:08:54.760699+0000","last_peered":"2024-03-28T02:08:54.760699+0000","last_clean":"2024-03-28T02:08:54.760699+0000","last_became_active":"2024-03-28T02:08:53.495442+0000","last_became_peered":"2024-03-28T02:08:53.495442+0000","last_unstale":"2024-03-28T02:08:54.760699+0000","last_undegraded":"2024-03-28T02:08:54.760699+0000","last_fullsized":"2024-03-28T02:08:54.760699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:32:33.318061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483338+0000","last_change":"2024-03-28T02:08:48.450376+0000","last_active":"2024-03-28T02:08:54.483338+0000","last_peered":"2024-03-28T02:08:54.483338+0000","last_clean":"2024-03-28T02:08:54.483338+0000","last_became_active":"2024-03-28T02:08:48.449978+0000","last_became_peered":"2024-03-28T02:08:48.449978+0000","last_unstale":"2024-03-28T02:08:54.483338+0000","last_undegraded":"2024-03-28T02:08:54.483338+0000","last_fullsized":"2024-03-28T02:08:54.483338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:33:56.284025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873674+0000","last_change":"2024-03-28T02:08:49.269011+0000","last_active":"2024-03-28T02:08:55.873674+0000","last_peered":"2024-03-28T02:08:55.873674+0000","last_clean":"2024-03-28T02:08:55.873674+0000","last_became_active":"2024-03-28T02:08:49.268141+0000","last_became_peered":"2024-03-28T02:08:49.268141+0000","last_unstale":"2024-03-28T02:08:55.873674+0000","last_undegraded":"2024-03-28T02:08:55.873674+0000","last_fullsized":"2024-03-28T02:08:55.873674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:54:32.708445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.733856+0000","last_change":"2024-03-28T02:08:53.501549+0000","last_active":"2024-03-28T02:08:55.733856+0000","last_peered":"2024-03-28T02:08:55.733856+0000","last_clean":"2024-03-28T02:08:55.733856+0000","last_became_active":"2024-03-28T02:08:53.501017+0000","last_became_peered":"2024-03-28T02:08:53.501017+0000","last_unstale":"2024-03-28T02:08:55.733856+0000","last_undegraded":"2024-03-28T02:08:55.733856+0000","last_fullsized":"2024-03-28T02:08:55.733856+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:33:50.346117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.611029+0000","last_change":"2024-03-28T02:08:48.454671+0000","last_active":"2024-03-28T02:08:55.611029+0000","last_peered":"2024-03-28T02:08:55.611029+0000","last_clean":"2024-03-28T02:08:55.611029+0000","last_became_active":"2024-03-28T02:08:48.454329+0000","last_became_peered":"2024-03-28T02:08:48.454329+0000","last_unstale":"2024-03-28T02:08:55.611029+0000","last_undegraded":"2024-03-28T02:08:55.611029+0000","last_fullsized":"2024-03-28T02:08:55.611029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:41:24.087559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873169+0000","last_change":"2024-03-28T02:08:49.259510+0000","last_active":"2024-03-28T02:08:55.873169+0000","last_peered":"2024-03-28T02:08:55.873169+0000","last_clean":"2024-03-28T02:08:55.873169+0000","last_became_active":"2024-03-28T02:08:49.259141+0000","last_became_peered":"2024-03-28T02:08:49.259141+0000","last_unstale":"2024-03-28T02:08:55.873169+0000","last_undegraded":"2024-03-28T02:08:55.873169+0000","last_fullsized":"2024-03-28T02:08:55.873169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:56:50.754576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610477+0000","last_change":"2024-03-28T02:08:53.488792+0000","last_active":"2024-03-28T02:08:55.610477+0000","last_peered":"2024-03-28T02:08:55.610477+0000","last_clean":"2024-03-28T02:08:55.610477+0000","last_became_active":"2024-03-28T02:08:53.488013+0000","last_became_peered":"2024-03-28T02:08:53.488013+0000","last_unstale":"2024-03-28T02:08:55.610477+0000","last_undegraded":"2024-03-28T02:08:55.610477+0000","last_fullsized":"2024-03-28T02:08:55.610477+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:39:52.006618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293634+0000","last_change":"2024-03-28T02:08:48.454609+0000","last_active":"2024-03-28T02:08:55.293634+0000","last_peered":"2024-03-28T02:08:55.293634+0000","last_clean":"2024-03-28T02:08:55.293634+0000","last_became_active":"2024-03-28T02:08:48.454297+0000","last_became_peered":"2024-03-28T02:08:48.454297+0000","last_unstale":"2024-03-28T02:08:55.293634+0000","last_undegraded":"2024-03-28T02:08:55.293634+0000","last_fullsized":"2024-03-28T02:08:55.293634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:31:42.819338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873773+0000","last_change":"2024-03-28T02:08:49.263700+0000","last_active":"2024-03-28T02:08:55.873773+0000","last_peered":"2024-03-28T02:08:55.873773+0000","last_clean":"2024-03-28T02:08:55.873773+0000","last_became_active":"2024-03-28T02:08:49.263073+0000","last_became_peered":"2024-03-28T02:08:49.263073+0000","last_unstale":"2024-03-28T02:08:55.873773+0000","last_undegraded":"2024-03-28T02:08:55.873773+0000","last_fullsized":"2024-03-28T02:08:55.873773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:34:19.158284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872835+0000","last_change":"2024-03-28T02:08:49.265572+0000","last_active":"2024-03-28T02:08:55.872835+0000","last_peered":"2024-03-28T02:08:55.872835+0000","last_clean":"2024-03-28T02:08:55.872835+0000","last_became_active":"2024-03-28T02:08:49.265202+0000","last_became_peered":"2024-03-28T02:08:49.265202+0000","last_unstale":"2024-03-28T02:08:55.872835+0000","last_undegraded":"2024-03-28T02:08:55.872835+0000","last_fullsized":"2024-03-28T02:08:55.872835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:00:05.296762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482974+0000","last_change":"2024-03-28T02:08:49.266529+0000","last_active":"2024-03-28T02:08:54.482974+0000","last_peered":"2024-03-28T02:08:54.482974+0000","last_clean":"2024-03-28T02:08:54.482974+0000","last_became_active":"2024-03-28T02:08:49.266162+0000","last_became_peered":"2024-03-28T02:08:49.266162+0000","last_unstale":"2024-03-28T02:08:54.482974+0000","last_undegraded":"2024-03-28T02:08:54.482974+0000","last_fullsized":"2024-03-28T02:08:54.482974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:31:40.117845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.482760+0000","last_change":"2024-03-28T02:08:49.264712+0000","last_active":"2024-03-28T02:08:54.482760+0000","last_peered":"2024-03-28T02:08:54.482760+0000","last_clean":"2024-03-28T02:08:54.482760+0000","last_became_active":"2024-03-28T02:08:49.264363+0000","last_became_peered":"2024-03-28T02:08:49.264363+0000","last_unstale":"2024-03-28T02:08:54.482760+0000","last_undegraded":"2024-03-28T02:08:54.482760+0000","last_fullsized":"2024-03-28T02:08:54.482760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:14:09.178725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.734001+0000","last_change":"2024-03-28T02:08:49.261713+0000","last_active":"2024-03-28T02:08:55.734001+0000","last_peered":"2024-03-28T02:08:55.734001+0000","last_clean":"2024-03-28T02:08:55.734001+0000","last_became_active":"2024-03-28T02:08:49.261311+0000","last_became_peered":"2024-03-28T02:08:49.261311+0000","last_unstale":"2024-03-28T02:08:55.734001+0000","last_undegraded":"2024-03-28T02:08:55.734001+0000","last_fullsized":"2024-03-28T02:08:55.734001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:56:54.801575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293500+0000","last_change":"2024-03-28T02:08:49.256674+0000","last_active":"2024-03-28T02:08:55.293500+0000","last_peered":"2024-03-28T02:08:55.293500+0000","last_clean":"2024-03-28T02:08:55.293500+0000","last_became_active":"2024-03-28T02:08:49.256360+0000","last_became_peered":"2024-03-28T02:08:49.256360+0000","last_unstale":"2024-03-28T02:08:55.293500+0000","last_undegraded":"2024-03-28T02:08:55.293500+0000","last_fullsized":"2024-03-28T02:08:55.293500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:44:14.817569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873468+0000","last_change":"2024-03-28T02:08:49.259829+0000","last_active":"2024-03-28T02:08:55.873468+0000","last_peered":"2024-03-28T02:08:55.873468+0000","last_clean":"2024-03-28T02:08:55.873468+0000","last_became_active":"2024-03-28T02:08:49.259419+0000","last_became_peered":"2024-03-28T02:08:49.259419+0000","last_unstale":"2024-03-28T02:08:55.873468+0000","last_undegraded":"2024-03-28T02:08:55.873468+0000","last_fullsized":"2024-03-28T02:08:55.873468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:49:46.023075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.483527+0000","last_change":"2024-03-28T02:08:49.263116+0000","last_active":"2024-03-28T02:08:54.483527+0000","last_peered":"2024-03-28T02:08:54.483527+0000","last_clean":"2024-03-28T02:08:54.483527+0000","last_became_active":"2024-03-28T02:08:49.262568+0000","last_became_peered":"2024-03-28T02:08:49.262568+0000","last_unstale":"2024-03-28T02:08:54.483527+0000","last_undegraded":"2024-03-28T02:08:54.483527+0000","last_fullsized":"2024-03-28T02:08:54.483527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:29:50.397737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873142+0000","last_change":"2024-03-28T02:08:49.261449+0000","last_active":"2024-03-28T02:08:55.873142+0000","last_peered":"2024-03-28T02:08:55.873142+0000","last_clean":"2024-03-28T02:08:55.873142+0000","last_became_active":"2024-03-28T02:08:49.261092+0000","last_became_peered":"2024-03-28T02:08:49.261092+0000","last_unstale":"2024-03-28T02:08:55.873142+0000","last_undegraded":"2024-03-28T02:08:55.873142+0000","last_fullsized":"2024-03-28T02:08:55.873142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:51:39.388852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.610776+0000","last_change":"2024-03-28T02:08:49.260711+0000","last_active":"2024-03-28T02:08:55.610776+0000","last_peered":"2024-03-28T02:08:55.610776+0000","last_clean":"2024-03-28T02:08:55.610776+0000","last_became_active":"2024-03-28T02:08:49.259358+0000","last_became_peered":"2024-03-28T02:08:49.259358+0000","last_unstale":"2024-03-28T02:08:55.610776+0000","last_undegraded":"2024-03-28T02:08:55.610776+0000","last_fullsized":"2024-03-28T02:08:55.610776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:27:09.084143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.872568+0000","last_change":"2024-03-28T02:08:49.267145+0000","last_active":"2024-03-28T02:08:55.872568+0000","last_peered":"2024-03-28T02:08:55.872568+0000","last_clean":"2024-03-28T02:08:55.872568+0000","last_became_active":"2024-03-28T02:08:49.266826+0000","last_became_peered":"2024-03-28T02:08:49.266826+0000","last_unstale":"2024-03-28T02:08:55.872568+0000","last_undegraded":"2024-03-28T02:08:55.872568+0000","last_fullsized":"2024-03-28T02:08:55.872568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:37:20.143596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:54.760661+0000","last_change":"2024-03-28T02:08:49.266752+0000","last_active":"2024-03-28T02:08:54.760661+0000","last_peered":"2024-03-28T02:08:54.760661+0000","last_clean":"2024-03-28T02:08:54.760661+0000","last_became_active":"2024-03-28T02:08:49.266436+0000","last_became_peered":"2024-03-28T02:08:49.266436+0000","last_unstale":"2024-03-28T02:08:54.760661+0000","last_undegraded":"2024-03-28T02:08:54.760661+0000","last_fullsized":"2024-03-28T02:08:54.760661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:52:26.578174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.293471+0000","last_change":"2024-03-28T02:08:49.258934+0000","last_active":"2024-03-28T02:08:55.293471+0000","last_peered":"2024-03-28T02:08:55.293471+0000","last_clean":"2024-03-28T02:08:55.293471+0000","last_became_active":"2024-03-28T02:08:49.258403+0000","last_became_peered":"2024-03-28T02:08:49.258403+0000","last_unstale":"2024-03-28T02:08:55.293471+0000","last_undegraded":"2024-03-28T02:08:55.293471+0000","last_fullsized":"2024-03-28T02:08:55.293471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:44:45.886136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T02:08:55.873307+0000","last_change":"2024-03-28T02:08:49.268456+0000","last_active":"2024-03-28T02:08:55.873307+0000","last_peered":"2024-03-28T02:08:55.873307+0000","last_clean":"2024-03-28T02:08:55.873307+0000","last_became_active":"2024-03-28T02:08:49.267346+0000","last_became_peered":"2024-03-28T02:08:49.267346+0000","last_unstale":"2024-03-28T02:08:55.873307+0000","last_undegraded":"2024-03-28T02:08:55.873307+0000","last_fullsized":"2024-03-28T02:08:55.873307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:16:47.157938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283904,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4730,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":68,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":16928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57,"ondisk_log_size":57,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1073003,"data_compressed":50774,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":477819,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":50774,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477909,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50774,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T02:09:12.331 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T02:09:12.332 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T02:09:12.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T02:09:12.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:12 smithi106 ceph-mon[31965]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:12.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:12 smithi106 ceph-mon[31965]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:12.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:12 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] up:active 2024-03-28T02:09:12.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:12 smithi106 ceph-mon[31965]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-28T02:09:12.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:12 smithi106 ceph-mon[31965]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T02:09:12.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:12 smithi106 ceph-mon[31965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T02:09:12.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:12 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2815885271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:09:12.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:12 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/69182963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:12.609 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:12.621 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:12 smithi178 ceph-mon[36521]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:12.621 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:12 smithi178 ceph-mon[36521]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:12 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] up:active 2024-03-28T02:09:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:12 smithi178 ceph-mon[36521]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-28T02:09:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:12 smithi178 ceph-mon[36521]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T02:09:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:12 smithi178 ceph-mon[36521]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T02:09:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:12 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2815885271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:09:12.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:12 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/69182963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:12.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:12 smithi153 ceph-mon[36228]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:12 smithi153 ceph-mon[36228]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:09:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:12 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] up:active 2024-03-28T02:09:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:12 smithi153 ceph-mon[36228]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-28T02:09:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:12 smithi153 ceph-mon[36228]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T02:09:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:12 smithi153 ceph-mon[36228]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T02:09:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:12 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2815885271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:09:12.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:12 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/69182963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:13.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:13 smithi106 ceph-mon[31965]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:09:13.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:13 smithi106 ceph-mon[31965]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:09:13.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:13 smithi106 ceph-mon[31965]: pgmap v304: 129 pgs: 129 active+clean; 587 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-03-28T02:09:13.621 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:13 smithi178 ceph-mon[36521]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:09:13.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:13 smithi178 ceph-mon[36521]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:09:13.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:13 smithi178 ceph-mon[36521]: pgmap v304: 129 pgs: 129 active+clean; 587 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-03-28T02:09:13.732 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:09:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b[67179]: 2024-03-28T02:09:13.417+0000 7f3a59697700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-28T02:09:13.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:13 smithi153 ceph-mon[36228]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:09:13.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:13 smithi153 ceph-mon[36228]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:09:13.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:13 smithi153 ceph-mon[36228]: pgmap v304: 129 pgs: 129 active+clean; 587 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-03-28T02:09:14.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.414+0000 7f041dc6c700 1 -- 172.21.15.106:0/1469551122 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418107ed0 msgr2=0x7f041810a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:14.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.414+0000 7f041dc6c700 1 --2- 172.21.15.106:0/1469551122 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418107ed0 0x7f041810a340 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f041000b0a0 tx=0x7f041000b3b0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.414+0000 7f041dc6c700 1 -- 172.21.15.106:0/1469551122 shutdown_connections 2024-03-28T02:09:14.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.414+0000 7f041dc6c700 1 --2- 172.21.15.106:0/1469551122 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418107ed0 0x7f041810a340 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.414+0000 7f041dc6c700 1 --2- 172.21.15.106:0/1469551122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0418102c70 0x7f0418105060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.414+0000 7f041dc6c700 1 --2- 172.21.15.106:0/1469551122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04181055a0 0x7f0418107990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.414+0000 7f041dc6c700 1 -- 172.21.15.106:0/1469551122 >> 172.21.15.106:0/1469551122 conn(0x7f04180fc680 msgr2=0x7f04180feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:14.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.414+0000 7f041dc6c700 1 -- 172.21.15.106:0/1469551122 shutdown_connections 2024-03-28T02:09:14.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.414+0000 7f041dc6c700 1 -- 172.21.15.106:0/1469551122 wait complete. 2024-03-28T02:09:14.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.415+0000 7f041dc6c700 1 Processor -- start 2024-03-28T02:09:14.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.415+0000 7f041dc6c700 1 -- start start 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.415+0000 7f041dc6c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418102c70 0x7f04181982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.415+0000 7f041dc6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04181055a0 0x7f04181987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.415+0000 7f041dc6c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0418107ed0 0x7f041819f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.415+0000 7f041dc6c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041810ddf0 con 0x7f0418102c70 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.415+0000 7f041dc6c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041810dc70 con 0x7f04181055a0 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.415+0000 7f041dc6c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041810df70 con 0x7f0418107ed0 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f04177fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418102c70 0x7f04181982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f04177fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418102c70 0x7f04181982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35884/0 (socket says 172.21.15.106:35884) 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0416ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04181055a0 0x7f04181987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f04177fe700 1 -- 172.21.15.106:0/3718680029 learned_addr learned my addr 172.21.15.106:0/3718680029 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0417fff700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0418107ed0 0x7f041819f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0416ffd700 1 -- 172.21.15.106:0/3718680029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0418107ed0 msgr2=0x7f041819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0416ffd700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0418107ed0 0x7f041819f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0416ffd700 1 -- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418102c70 msgr2=0x7f04181982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0416ffd700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418102c70 0x7f04181982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0416ffd700 1 -- 172.21.15.106:0/3718680029 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041819ff10 con 0x7f04181055a0 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f04177fe700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418102c70 0x7f04181982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0416ffd700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04181055a0 0x7f04181987f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f040c00d750 tx=0x7f040c00da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0414ff9700 1 -- 172.21.15.106:0/3718680029 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c014030 con 0x7f04181055a0 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04181a0200 con 0x7f04181055a0 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0414ff9700 1 -- 172.21.15.106:0/3718680029 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f040c004500 con 0x7f04181055a0 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f0414ff9700 1 -- 172.21.15.106:0/3718680029 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c0054b0 con 0x7f04181055a0 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.416+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04181a0790 con 0x7f04181055a0 2024-03-28T02:09:14.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.417+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04180506a0 con 0x7f04181055a0 2024-03-28T02:09:14.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.418+0000 7f0414ff9700 1 -- 172.21.15.106:0/3718680029 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f040c004750 con 0x7f04181055a0 2024-03-28T02:09:14.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.419+0000 7f0414ff9700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f04000a29d0 0x7f04000a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:14.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.419+0000 7f04177fe700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f04000a29d0 0x7f04000a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:14.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.419+0000 7f0414ff9700 1 -- 172.21.15.106:0/3718680029 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f040c0ce9b0 con 0x7f04181055a0 2024-03-28T02:09:14.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.419+0000 7f04177fe700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f04000a29d0 0x7f04000a4e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0408011550 tx=0x7f0408009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:14.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.426+0000 7f0414ff9700 1 -- 172.21.15.106:0/3718680029 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f040c09b6b0 con 0x7f04181055a0 2024-03-28T02:09:14.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.706+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f041802da30 con 0x7f04181055a0 2024-03-28T02:09:14.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.711+0000 7f0414ff9700 1 -- 172.21.15.106:0/3718680029 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f040c09afb0 con 0x7f04181055a0 2024-03-28T02:09:14.704 INFO:teuthology.orchestra.run.smithi106.stdout:[client.0] 2024-03-28T02:09:14.704 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQBK0QRm2B07KhAAqrv0ij3RSIZVFHFx23j0RQ== 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.713+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f04000a29d0 msgr2=0x7f04000a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.713+0000 7f041dc6c700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f04000a29d0 0x7f04000a4e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0408011550 tx=0x7f0408009260 comp rx=0 tx=0).stop 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.713+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04181055a0 msgr2=0x7f04181987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.713+0000 7f041dc6c700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04181055a0 0x7f04181987f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f040c00d750 tx=0x7f040c00da60 comp rx=0 tx=0).stop 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.714+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 shutdown_connections 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.714+0000 7f041dc6c700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0418107ed0 0x7f041819f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.714+0000 7f041dc6c700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f04000a29d0 0x7f04000a4e90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.714+0000 7f041dc6c700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0418102c70 0x7f04181982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.714+0000 7f041dc6c700 1 --2- 172.21.15.106:0/3718680029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04181055a0 0x7f04181987f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.714+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 >> 172.21.15.106:0/3718680029 conn(0x7f04180fc680 msgr2=0x7f04181035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.714+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 shutdown_connections 2024-03-28T02:09:14.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:14.714+0000 7f041dc6c700 1 -- 172.21.15.106:0/3718680029 wait complete. 2024-03-28T02:09:15.513 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:09:15.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T02:09:15.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T02:09:15.559 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T02:09:15.559 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T02:09:15.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T02:09:15.718 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:15 smithi178 ceph-mon[36521]: pgmap v305: 129 pgs: 129 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-03-28T02:09:15.718 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:15 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3718680029' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T02:09:15.718 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:15 smithi178 ceph-mon[36521]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T02:09:15.718 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:15 smithi178 ceph-mon[36521]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T02:09:15.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:15 smithi153 ceph-mon[36228]: pgmap v305: 129 pgs: 129 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-03-28T02:09:15.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:15 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3718680029' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T02:09:15.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:15 smithi153 ceph-mon[36228]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T02:09:15.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:15 smithi153 ceph-mon[36228]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T02:09:15.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:15 smithi106 ceph-mon[31965]: pgmap v305: 129 pgs: 129 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-03-28T02:09:15.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:15 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3718680029' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T02:09:15.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:15 smithi106 ceph-mon[31965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T02:09:15.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:15 smithi106 ceph-mon[31965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T02:09:15.837 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:17.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:17 smithi106 ceph-mon[31965]: pgmap v306: 129 pgs: 129 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.4 KiB/s wr, 22 op/s 2024-03-28T02:09:17.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 -- 172.21.15.106:0/2648842479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7441081c0 msgr2=0x7fb7441085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:17.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/2648842479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7441081c0 0x7fb7441085a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb7400099c0 tx=0x7fb740009cd0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 -- 172.21.15.106:0/2648842479 shutdown_connections 2024-03-28T02:09:17.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/2648842479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb744102ba0 0x7fb74410efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/2648842479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7441081c0 0x7fb7441085a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/2648842479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7441021e0 0x7fb744102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 -- 172.21.15.106:0/2648842479 >> 172.21.15.106:0/2648842479 conn(0x7fb7440792c0 msgr2=0x7fb7440796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:17.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 -- 172.21.15.106:0/2648842479 shutdown_connections 2024-03-28T02:09:17.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 -- 172.21.15.106:0/2648842479 wait complete. 2024-03-28T02:09:17.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 Processor -- start 2024-03-28T02:09:17.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 -- start start 2024-03-28T02:09:17.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.691+0000 7fb74c0ba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7441021e0 0x7fb744074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:17.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb74c0ba700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb744102ba0 0x7fb7440749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:17.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb74c0ba700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7441081c0 0x7fb744074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:17.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb74c0ba700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7441124c0 con 0x7fb744102ba0 2024-03-28T02:09:17.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb74c0ba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744112340 con 0x7fb7441021e0 2024-03-28T02:09:17.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb74c0ba700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744112640 con 0x7fb7441081c0 2024-03-28T02:09:17.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749655700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb744102ba0 0x7fb7440749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749e56700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7441021e0 0x7fb744074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749655700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb744102ba0 0x7fb7440749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42912/0 (socket says 172.21.15.106:42912) 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb74a657700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7441081c0 0x7fb744074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749655700 1 -- 172.21.15.106:0/3268190022 learned_addr learned my addr 172.21.15.106:0/3268190022 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749e56700 1 -- 172.21.15.106:0/3268190022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7441081c0 msgr2=0x7fb744074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749e56700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7441081c0 0x7fb744074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749e56700 1 -- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb744102ba0 msgr2=0x7fb7440749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749e56700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb744102ba0 0x7fb7440749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749e56700 1 -- 172.21.15.106:0/3268190022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7441a6ec0 con 0x7fb7441021e0 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749655700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb744102ba0 0x7fb7440749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb74a657700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7441081c0 0x7fb744074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb749e56700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7441021e0 0x7fb744074490 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb7400098e0 tx=0x7fb74000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb73affd700 1 -- 172.21.15.106:0/3268190022 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740027070 con 0x7fb7441021e0 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7441a7150 con 0x7fb7441021e0 2024-03-28T02:09:17.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb73affd700 1 -- 172.21.15.106:0/3268190022 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb740022470 con 0x7fb7441021e0 2024-03-28T02:09:17.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7441a7600 con 0x7fb7441021e0 2024-03-28T02:09:17.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.692+0000 7fb73affd700 1 -- 172.21.15.106:0/3268190022 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740009e60 con 0x7fb7441021e0 2024-03-28T02:09:17.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.693+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb744068e50 con 0x7fb7441021e0 2024-03-28T02:09:17.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.695+0000 7fb73affd700 1 -- 172.21.15.106:0/3268190022 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb74002a070 con 0x7fb7441021e0 2024-03-28T02:09:17.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.695+0000 7fb73affd700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb7300a2920 0x7fb7300a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:17.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.695+0000 7fb749655700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb7300a2920 0x7fb7300a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:17.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.695+0000 7fb73affd700 1 -- 172.21.15.106:0/3268190022 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb7400cfb70 con 0x7fb7441021e0 2024-03-28T02:09:17.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.696+0000 7fb749655700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb7300a2920 0x7fb7300a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb7440798a0 tx=0x7fb734009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:17.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.699+0000 7fb73affd700 1 -- 172.21.15.106:0/3268190022 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb740013070 con 0x7fb7441021e0 2024-03-28T02:09:17.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:17 smithi153 ceph-mon[36228]: pgmap v306: 129 pgs: 129 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.4 KiB/s wr, 22 op/s 2024-03-28T02:09:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:17 smithi178 ceph-mon[36521]: pgmap v306: 129 pgs: 129 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.4 KiB/s wr, 22 op/s 2024-03-28T02:09:17.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.949+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb74402da30 con 0x7fb7441021e0 2024-03-28T02:09:17.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.957+0000 7fb73affd700 1 -- 172.21.15.106:0/3268190022 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262669 (secure 0 0 0) 0x7fb74009ca20 con 0x7fb7441021e0 2024-03-28T02:09:17.953 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:17.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.961+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb7300a2920 msgr2=0x7fb7300a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:17.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.961+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb7300a2920 0x7fb7300a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb7440798a0 tx=0x7fb734009260 comp rx=0 tx=0).stop 2024-03-28T02:09:17.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.961+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7441021e0 msgr2=0x7fb744074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:17.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.961+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7441021e0 0x7fb744074490 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb7400098e0 tx=0x7fb74000f690 comp rx=0 tx=0).stop 2024-03-28T02:09:17.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.961+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 shutdown_connections 2024-03-28T02:09:17.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.961+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7441081c0 0x7fb744074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.961+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7441021e0 0x7fb744074490 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.961+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb7300a2920 0x7fb7300a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.961+0000 7fb74c0ba700 1 --2- 172.21.15.106:0/3268190022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb744102ba0 0x7fb7440749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:17.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.962+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 >> 172.21.15.106:0/3268190022 conn(0x7fb7440792c0 msgr2=0x7fb744100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:17.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.962+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 shutdown_connections 2024-03-28T02:09:17.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:17.962+0000 7fb74c0ba700 1 -- 172.21.15.106:0/3268190022 wait complete. 2024-03-28T02:09:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:18 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3268190022' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T02:09:18.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:18 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3268190022' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T02:09:18.790 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3501119323},{"type":"v1","addr":"172.21.15.106:6801","nonce":3501119323}]},"active_addr":"172.21.15.106:6801/3501119323","active_change":"2024-03-28T02:01:42.268091+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":14214,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.106:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":2664246824}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3655727768}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":899620795}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":2601893696}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":1370688400}]}]} 2024-03-28T02:09:18.794 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T02:09:18.794 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T02:09:18.795 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T02:09:18.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:18 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3268190022' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T02:09:19.034 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:19.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:19 smithi153 ceph-mon[36228]: pgmap v307: 129 pgs: 129 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 9.4 KiB/s wr, 22 op/s 2024-03-28T02:09:19.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:19 smithi106 ceph-mon[31965]: pgmap v307: 129 pgs: 129 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 9.4 KiB/s wr, 22 op/s 2024-03-28T02:09:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:19 smithi178 ceph-mon[36521]: pgmap v307: 129 pgs: 129 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 9.4 KiB/s wr, 22 op/s 2024-03-28T02:09:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.894+0000 7f01f1978700 1 -- 172.21.15.106:0/2174074932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ec102ba0 msgr2=0x7f01ec10efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.894+0000 7f01f1978700 1 --2- 172.21.15.106:0/2174074932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ec102ba0 0x7f01ec10efe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009fc0 tx=0x7f01e400b330 comp rx=0 tx=0).stop 2024-03-28T02:09:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.894+0000 7f01f1978700 1 -- 172.21.15.106:0/2174074932 shutdown_connections 2024-03-28T02:09:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.894+0000 7f01f1978700 1 --2- 172.21.15.106:0/2174074932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ec102ba0 0x7f01ec10efe0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.894+0000 7f01f1978700 1 --2- 172.21.15.106:0/2174074932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ec1081c0 0x7f01ec1085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.894+0000 7f01f1978700 1 --2- 172.21.15.106:0/2174074932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01ec1021e0 0x7f01ec102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.894+0000 7f01f1978700 1 -- 172.21.15.106:0/2174074932 >> 172.21.15.106:0/2174074932 conn(0x7f01ec0792c0 msgr2=0x7f01ec0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.894+0000 7f01f1978700 1 -- 172.21.15.106:0/2174074932 shutdown_connections 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.894+0000 7f01f1978700 1 -- 172.21.15.106:0/2174074932 wait complete. 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01f1978700 1 Processor -- start 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01f1978700 1 -- start start 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01f1978700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ec1021e0 0x7f01ec19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01f1978700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01ec102ba0 0x7f01ec19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01f1978700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ec1081c0 0x7f01ec198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01f1978700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec112470 con 0x7f01ec1081c0 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01f1978700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec1122f0 con 0x7f01ec1021e0 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01f1978700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec1125f0 con 0x7f01ec102ba0 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01eb7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ec1081c0 0x7f01ec198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.895+0000 7f01eaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ec1021e0 0x7f01ec19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01eb7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ec1081c0 0x7f01ec198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42926/0 (socket says 172.21.15.106:42926) 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01eb7fe700 1 -- 172.21.15.106:0/541351069 learned_addr learned my addr 172.21.15.106:0/541351069 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01ea7fc700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01ec102ba0 0x7f01ec19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01eb7fe700 1 -- 172.21.15.106:0/541351069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01ec102ba0 msgr2=0x7f01ec19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01eb7fe700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01ec102ba0 0x7f01ec19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01eb7fe700 1 -- 172.21.15.106:0/541351069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ec1021e0 msgr2=0x7f01ec19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01eb7fe700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ec1021e0 0x7f01ec19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01eb7fe700 1 -- 172.21.15.106:0/541351069 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec198f30 con 0x7f01ec1081c0 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01eb7fe700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ec1081c0 0x7f01ec198650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009780 tx=0x7f01e4004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01f0976700 1 -- 172.21.15.106:0/541351069 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4021070 con 0x7f01ec1081c0 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01f0976700 1 -- 172.21.15.106:0/541351069 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01e4007ac0 con 0x7f01ec1081c0 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec1991c0 con 0x7f01ec1081c0 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01f0976700 1 -- 172.21.15.106:0/541351069 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40052a0 con 0x7f01ec1081c0 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.896+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec1a5330 con 0x7f01ec1081c0 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.897+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec103870 con 0x7f01ec1081c0 2024-03-28T02:09:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.898+0000 7f01f0976700 1 -- 172.21.15.106:0/541351069 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f01e4024070 con 0x7f01ec1081c0 2024-03-28T02:09:20.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.899+0000 7f01f0976700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f01d40a2920 0x7f01d40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:20.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.899+0000 7f01eaffd700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f01d40a2920 0x7f01d40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:20.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.899+0000 7f01f0976700 1 -- 172.21.15.106:0/541351069 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f01e40cffa0 con 0x7f01ec1081c0 2024-03-28T02:09:20.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.899+0000 7f01eaffd700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f01d40a2920 0x7f01d40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009c60 tx=0x7f01e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:20.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:20.902+0000 7f01f0976700 1 -- 172.21.15.106:0/541351069 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f01e409cd30 con 0x7f01ec1081c0 2024-03-28T02:09:21.089 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.096+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01ec02da30 con 0x7f01ec1081c0 2024-03-28T02:09:21.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.098+0000 7f01f0976700 1 -- 172.21.15.106:0/541351069 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7f01e409c630 con 0x7f01ec1081c0 2024-03-28T02:09:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:21.092 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:09:21.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f01d40a2920 msgr2=0x7f01d40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:21.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f01d40a2920 0x7f01d40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009c60 tx=0x7f01e0009260 comp rx=0 tx=0).stop 2024-03-28T02:09:21.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ec1081c0 msgr2=0x7f01ec198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:21.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ec1081c0 0x7f01ec198650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009780 tx=0x7f01e4004f20 comp rx=0 tx=0).stop 2024-03-28T02:09:21.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 shutdown_connections 2024-03-28T02:09:21.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f01d40a2920 0x7f01d40a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:21.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ec1081c0 0x7f01ec198650 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:21.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ec1021e0 0x7f01ec19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:21.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 --2- 172.21.15.106:0/541351069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01ec102ba0 0x7f01ec19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:21.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 >> 172.21.15.106:0/541351069 conn(0x7f01ec0792c0 msgr2=0x7f01ec100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:21.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 shutdown_connections 2024-03-28T02:09:21.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:21.100+0000 7f01f1978700 1 -- 172.21.15.106:0/541351069 wait complete. 2024-03-28T02:09:21.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:21 smithi153 ceph-mon[36228]: pgmap v308: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 25 op/s 2024-03-28T02:09:21.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:21 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/541351069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:21 smithi106 ceph-mon[31965]: pgmap v308: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 25 op/s 2024-03-28T02:09:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:21 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/541351069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:21 smithi178 ceph-mon[36521]: pgmap v308: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 25 op/s 2024-03-28T02:09:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:21 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/541351069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:21.927 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T02:09:21.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T02:09:22.164 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:23.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:23 smithi106 ceph-mon[31965]: pgmap v309: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 25 op/s 2024-03-28T02:09:23.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:23 smithi153 ceph-mon[36228]: pgmap v309: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 25 op/s 2024-03-28T02:09:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:23 smithi178 ceph-mon[36521]: pgmap v309: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 25 op/s 2024-03-28T02:09:23.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.977+0000 7ff8ffa79700 1 -- 172.21.15.106:0/1797224995 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 msgr2=0x7ff8f81085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:23.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.977+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/1797224995 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 0x7ff8f81085a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80099c0 tx=0x7ff8e8009cd0 comp rx=0 tx=0).stop 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.977+0000 7ff8ffa79700 1 -- 172.21.15.106:0/1797224995 shutdown_connections 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.977+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/1797224995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f8102ba0 0x7ff8f810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.977+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/1797224995 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 0x7ff8f81085a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.977+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/1797224995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f81021e0 0x7ff8f8102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.977+0000 7ff8ffa79700 1 -- 172.21.15.106:0/1797224995 >> 172.21.15.106:0/1797224995 conn(0x7ff8f80792c0 msgr2=0x7ff8f80796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.977+0000 7ff8ffa79700 1 -- 172.21.15.106:0/1797224995 shutdown_connections 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.977+0000 7ff8ffa79700 1 -- 172.21.15.106:0/1797224995 wait complete. 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8ffa79700 1 Processor -- start 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8ffa79700 1 -- start start 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8ffa79700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f81021e0 0x7ff8f819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8ffa79700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8102ba0 0x7ff8f819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8ffa79700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 0x7ff8f8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8ffa79700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8112440 con 0x7ff8f81081c0 2024-03-28T02:09:23.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8ffa79700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f81122c0 con 0x7ff8f8102ba0 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8ffa79700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f81125c0 con 0x7ff8f81021e0 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fe016700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 0x7ff8f8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fd815700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f81021e0 0x7ff8f819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fe016700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 0x7ff8f8198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42936/0 (socket says 172.21.15.106:42936) 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fd815700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f81021e0 0x7ff8f819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:43834/0 (socket says 172.21.15.106:43834) 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fe016700 1 -- 172.21.15.106:0/2345660828 learned_addr learned my addr 172.21.15.106:0/2345660828 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fd014700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8102ba0 0x7ff8f819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fe016700 1 -- 172.21.15.106:0/2345660828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f81021e0 msgr2=0x7ff8f819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fe016700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f81021e0 0x7ff8f819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fe016700 1 -- 172.21.15.106:0/2345660828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8102ba0 msgr2=0x7ff8f819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fe016700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8102ba0 0x7ff8f819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:23.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.978+0000 7ff8fe016700 1 -- 172.21.15.106:0/2345660828 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f8199000 con 0x7ff8f81081c0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.979+0000 7ff8fe016700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 0x7ff8f8198720 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000ef30 tx=0x7ff8f000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.979+0000 7ff8eeffd700 1 -- 172.21.15.106:0/2345660828 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f00092e0 con 0x7ff8f81081c0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.979+0000 7ff8eeffd700 1 -- 172.21.15.106:0/2345660828 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8f0004bb0 con 0x7ff8f81081c0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.979+0000 7ff8eeffd700 1 -- 172.21.15.106:0/2345660828 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0004280 con 0x7ff8f81081c0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.979+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f8199290 con 0x7ff8f81081c0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.979+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f81a55d0 con 0x7ff8f81081c0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.980+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f8103870 con 0x7ff8f81081c0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.981+0000 7ff8eeffd700 1 -- 172.21.15.106:0/2345660828 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ff8f001b030 con 0x7ff8f81081c0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.982+0000 7ff8eeffd700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff8e40a29a0 0x7ff8e40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.982+0000 7ff8fd815700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff8e40a29a0 0x7ff8e40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.982+0000 7ff8eeffd700 1 -- 172.21.15.106:0/2345660828 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff8f00cf1a0 con 0x7ff8f81081c0 2024-03-28T02:09:23.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.982+0000 7ff8fd815700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff8e40a29a0 0x7ff8e40a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8000c00 tx=0x7ff8e8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:23.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:23.986+0000 7ff8eeffd700 1 -- 172.21.15.106:0/2345660828 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff8f0022020 con 0x7ff8f81081c0 2024-03-28T02:09:24.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.180+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8f8199c50 con 0x7ff8f81081c0 2024-03-28T02:09:24.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.181+0000 7ff8eeffd700 1 -- 172.21.15.106:0/2345660828 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24206 (secure 0 0 0) 0x7ff8f0010040 con 0x7ff8f81081c0 2024-03-28T02:09:24.175 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:09:24.175 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":89,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:08:53.828139+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:09:24.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff8e40a29a0 msgr2=0x7ff8e40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:24.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff8e40a29a0 0x7ff8e40a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8000c00 tx=0x7ff8e8005fb0 comp rx=0 tx=0).stop 2024-03-28T02:09:24.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 msgr2=0x7ff8f8198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:24.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 0x7ff8f8198720 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000ef30 tx=0x7ff8f000c3a0 comp rx=0 tx=0).stop 2024-03-28T02:09:24.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 shutdown_connections 2024-03-28T02:09:24.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff8e40a29a0 0x7ff8e40a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:24.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f81081c0 0x7ff8f8198720 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:24.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f81021e0 0x7ff8f819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:24.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 --2- 172.21.15.106:0/2345660828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8102ba0 0x7ff8f819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:24.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 >> 172.21.15.106:0/2345660828 conn(0x7ff8f80792c0 msgr2=0x7ff8f8100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:24.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 shutdown_connections 2024-03-28T02:09:24.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:24.184+0000 7ff8ffa79700 1 -- 172.21.15.106:0/2345660828 wait complete. 2024-03-28T02:09:24.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:24 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2345660828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:24.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:24 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2345660828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:24.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:24 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2345660828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:09:25.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T02:09:25.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T02:09:25.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T02:09:25.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T02:09:25.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T02:09:25.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T02:09:25.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T02:09:25.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T02:09:25.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T02:09:25.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T02:09:25.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T02:09:25.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T02:09:25.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:25 smithi153 ceph-mon[36228]: pgmap v310: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 15 op/s 2024-03-28T02:09:25.785 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:25.790 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:25.793 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:25.799 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:25.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:25 smithi106 ceph-mon[31965]: pgmap v310: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 15 op/s 2024-03-28T02:09:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:25 smithi178 ceph-mon[36521]: pgmap v310: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 15 op/s 2024-03-28T02:09:27.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:27 smithi153 ceph-mon[36228]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-28T02:09:27.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:27 smithi106 ceph-mon[31965]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-28T02:09:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:27 smithi178 ceph-mon[36521]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-28T02:09:29.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:29 smithi153 ceph-mon[36228]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T02:09:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:29 smithi106 ceph-mon[31965]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T02:09:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:29 smithi178 ceph-mon[36521]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T02:09:31.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.212+0000 7f313b4a9700 1 -- 172.21.15.106:0/2701778708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3134102590 msgr2=0x7f313410df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:31.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.212+0000 7f313b4a9700 1 --2- 172.21.15.106:0/2701778708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3134102590 0x7f313410df60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f312c00b0a0 tx=0x7f312c00b3b0 comp rx=0 tx=0).stop 2024-03-28T02:09:31.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.212+0000 7f313b4a9700 1 -- 172.21.15.106:0/2701778708 shutdown_connections 2024-03-28T02:09:31.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.212+0000 7f313b4a9700 1 --2- 172.21.15.106:0/2701778708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3134102590 0x7f313410df60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:31.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.212+0000 7f313b4a9700 1 --2- 172.21.15.106:0/2701778708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31341060c0 0x7f31341064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:31.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.212+0000 7f313b4a9700 1 --2- 172.21.15.106:0/2701778708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3134101bd0 0x7f3134102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:31.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.212+0000 7f313b4a9700 1 -- 172.21.15.106:0/2701778708 >> 172.21.15.106:0/2701778708 conn(0x7f31340763c0 msgr2=0x7f31340767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:31.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.212+0000 7f313b4a9700 1 -- 172.21.15.106:0/2701778708 shutdown_connections 2024-03-28T02:09:31.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.212+0000 7f313b4a9700 1 -- 172.21.15.106:0/2701778708 wait complete. 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.213+0000 7f313b4a9700 1 Processor -- start 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.213+0000 7f313b4a9700 1 -- start start 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.213+0000 7f313b4a9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3134101bd0 0x7f313419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.213+0000 7f313b4a9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3134102590 0x7f313419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.213+0000 7f313b4a9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31341060c0 0x7f3134198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.213+0000 7f313b4a9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31341113c0 con 0x7f3134101bd0 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.213+0000 7f313b4a9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3134111240 con 0x7f3134102590 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.213+0000 7f313b4a9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3134111540 con 0x7f31341060c0 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3138a44700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3134102590 0x7f313419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3138a44700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3134102590 0x7f313419eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:55694/0 (socket says 172.21.15.106:55694) 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3139a46700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31341060c0 0x7f3134198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3138a44700 1 -- 172.21.15.106:0/69809214 learned_addr learned my addr 172.21.15.106:0/69809214 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:31.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3139245700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3134101bd0 0x7f313419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:31.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3138a44700 1 -- 172.21.15.106:0/69809214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31341060c0 msgr2=0x7f3134198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:31.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3138a44700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31341060c0 0x7f3134198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:31.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3138a44700 1 -- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3134101bd0 msgr2=0x7f313419e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:31.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3138a44700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3134101bd0 0x7f313419e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:31.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3138a44700 1 -- 172.21.15.106:0/69809214 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3134198f70 con 0x7f3134102590 2024-03-28T02:09:31.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.214+0000 7f3138a44700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3134102590 0x7f313419eb00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f312400eb30 tx=0x7f312400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:31.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.215+0000 7f312a7fc700 1 -- 172.21.15.106:0/69809214 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312400cc90 con 0x7f3134102590 2024-03-28T02:09:31.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.215+0000 7f312a7fc700 1 -- 172.21.15.106:0/69809214 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3124004500 con 0x7f3134102590 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.215+0000 7f312a7fc700 1 -- 172.21.15.106:0/69809214 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124010430 con 0x7f3134102590 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.215+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3134199260 con 0x7f3134102590 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.215+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31341a5370 con 0x7f3134102590 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.217+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3134100940 con 0x7f3134102590 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.218+0000 7f312a7fc700 1 -- 172.21.15.106:0/69809214 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f3124004020 con 0x7f3134102590 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.218+0000 7f312a7fc700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31200a2a70 0x7f31200a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.218+0000 7f3139245700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31200a2a70 0x7f31200a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.218+0000 7f312a7fc700 1 -- 172.21.15.106:0/69809214 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f31240cf080 con 0x7f3134102590 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.219+0000 7f312a7fc700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] conn(0x7f31200ad730 0x7f31200afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.219+0000 7f312a7fc700 1 -- 172.21.15.106:0/69809214 --> [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31200b0240 con 0x7f31200ad730 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.219+0000 7f312a7fc700 1 -- 172.21.15.106:0/69809214 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31240d6050 con 0x7f3134102590 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.219+0000 7f3139a46700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] conn(0x7f31200ad730 0x7f31200afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:31.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.219+0000 7f3139245700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31200a2a70 0x7f31200a4f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3130009c60 tx=0x7f3130009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.219+0000 7f3139a46700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] conn(0x7f31200ad730 0x7f31200afb70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.220+0000 7f312a7fc700 1 -- 172.21.15.106:0/69809214 <== osd.3 v2:172.21.15.106:6810/3263525761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f31200b0240 con 0x7f31200ad730 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.237+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 --> [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f313406a4a0 con 0x7f31200ad730 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f312a7fc700 1 -- 172.21.15.106:0/69809214 <== osd.3 v2:172.21.15.106:6810/3263525761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f313406a4a0 con 0x7f31200ad730 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] conn(0x7f31200ad730 msgr2=0x7f31200afb70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] conn(0x7f31200ad730 0x7f31200afb70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31200a2a70 msgr2=0x7f31200a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31200a2a70 0x7f31200a4f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3130009c60 tx=0x7f3130009260 comp rx=0 tx=0).stop 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3134102590 msgr2=0x7f313419eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 --2- 172.21.15.106:0/69809214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3134102590 0x7f313419eb00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f312400eb30 tx=0x7f312400ee40 comp rx=0 tx=0).stop 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f3139a46700 1 -- 172.21.15.106:0/69809214 reap_dead start 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 shutdown_connections 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 >> 172.21.15.106:0/69809214 conn(0x7f31340763c0 msgr2=0x7f31341053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 shutdown_connections 2024-03-28T02:09:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:29.238+0000 7f313b4a9700 1 -- 172.21.15.106:0/69809214 wait complete. 2024-03-28T02:09:31.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:31 smithi153 ceph-mon[36228]: pgmap v313: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-28T02:09:31.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:31 smithi106 ceph-mon[31965]: pgmap v313: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-28T02:09:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:31 smithi178 ceph-mon[36521]: pgmap v313: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-28T02:09:32.104 INFO:teuthology.orchestra.run.smithi106.stdout:107374182462 2024-03-28T02:09:32.104 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T02:09:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.299+0000 7f58978cb700 1 -- 172.21.15.106:0/2066817101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5890101070 msgr2=0x7f58901014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.299+0000 7f58978cb700 1 --2- 172.21.15.106:0/2066817101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5890101070 0x7f58901014f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5880009990 tx=0x7f5880009ca0 comp rx=0 tx=0).stop 2024-03-28T02:09:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.299+0000 7f58978cb700 1 -- 172.21.15.106:0/2066817101 shutdown_connections 2024-03-28T02:09:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.299+0000 7f58978cb700 1 --2- 172.21.15.106:0/2066817101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5890101a30 0x7f589010ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.299+0000 7f58978cb700 1 --2- 172.21.15.106:0/2066817101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5890107120 0x7f5890107500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.299+0000 7f58978cb700 1 --2- 172.21.15.106:0/2066817101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5890101070 0x7f58901014f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.299+0000 7f58978cb700 1 -- 172.21.15.106:0/2066817101 >> 172.21.15.106:0/2066817101 conn(0x7f58900780a0 msgr2=0x7f58900784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.299+0000 7f58978cb700 1 -- 172.21.15.106:0/2066817101 shutdown_connections 2024-03-28T02:09:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.299+0000 7f58978cb700 1 -- 172.21.15.106:0/2066817101 wait complete. 2024-03-28T02:09:32.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.300+0000 7f58978cb700 1 Processor -- start 2024-03-28T02:09:32.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.300+0000 7f58978cb700 1 -- start start 2024-03-28T02:09:32.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.300+0000 7f58978cb700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5890101070 0x7f58901996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:32.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.300+0000 7f58978cb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5890101a30 0x7f5890199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:32.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.300+0000 7f58978cb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5890107120 0x7f58901a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:32.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5894e66700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5890101a30 0x7f5890199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895e68700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5890107120 0x7f58901a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895667700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5890101070 0x7f58901996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895e68700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5890107120 0x7f58901a4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:55716/0 (socket says 172.21.15.106:55716) 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895667700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5890101070 0x7f58901996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:39838/0 (socket says 172.21.15.106:39838) 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5894e66700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5890101a30 0x7f5890199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52274/0 (socket says 172.21.15.106:52274) 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895e68700 1 -- 172.21.15.106:0/445927101 learned_addr learned my addr 172.21.15.106:0/445927101 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.300+0000 7f58978cb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890111330 con 0x7f5890101a30 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58901111b0 con 0x7f5890107120 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58901114b0 con 0x7f5890101070 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895e68700 1 -- 172.21.15.106:0/445927101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5890101070 msgr2=0x7f58901996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895e68700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5890101070 0x7f58901996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895e68700 1 -- 172.21.15.106:0/445927101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5890101a30 msgr2=0x7f5890199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895e68700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5890101a30 0x7f5890199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895e68700 1 -- 172.21.15.106:0/445927101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58901a4c70 con 0x7f5890107120 2024-03-28T02:09:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895667700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5890101070 0x7f58901996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:32.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.301+0000 7f5895e68700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5890107120 0x7f58901a4730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f588800ef30 tx=0x7f588800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:32.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.302+0000 7f58867fc700 1 -- 172.21.15.106:0/445927101 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58880092e0 con 0x7f5890107120 2024-03-28T02:09:32.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.302+0000 7f58867fc700 1 -- 172.21.15.106:0/445927101 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5888004bb0 con 0x7f5890107120 2024-03-28T02:09:32.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.302+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58901a4ea0 con 0x7f5890107120 2024-03-28T02:09:32.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.302+0000 7f58867fc700 1 -- 172.21.15.106:0/445927101 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888004280 con 0x7f5890107120 2024-03-28T02:09:32.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.302+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58901a5300 con 0x7f5890107120 2024-03-28T02:09:32.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.303+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f58900628e0 con 0x7f5890107120 2024-03-28T02:09:32.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.304+0000 7f58867fc700 1 -- 172.21.15.106:0/445927101 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f588801b030 con 0x7f5890107120 2024-03-28T02:09:32.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.305+0000 7f58867fc700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f587c0a29a0 0x7f587c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:32.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.305+0000 7f58867fc700 1 -- 172.21.15.106:0/445927101 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f58880cf500 con 0x7f5890107120 2024-03-28T02:09:32.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.305+0000 7f5895667700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f587c0a29a0 0x7f587c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:32.299 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.305+0000 7f58867fc700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] conn(0x7f587c0ad660 0x7f587c0afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:32.299 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.305+0000 7f58867fc700 1 -- 172.21.15.106:0/445927101 --> [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5888022e50 con 0x7f587c0ad660 2024-03-28T02:09:32.299 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.305+0000 7f58867fc700 1 -- 172.21.15.106:0/445927101 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58880cf950 con 0x7f5890107120 2024-03-28T02:09:32.299 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.305+0000 7f5894e66700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] conn(0x7f587c0ad660 0x7f587c0afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:32.299 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.305+0000 7f5895667700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f587c0a29a0 0x7f587c0a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f588c009fd0 tx=0x7f588c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:32.299 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.306+0000 7f5894e66700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] conn(0x7f587c0ad660 0x7f587c0afaa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:32.299 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.306+0000 7f58867fc700 1 -- 172.21.15.106:0/445927101 <== osd.10 v2:172.21.15.153:6824/1470813442 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5888022e50 con 0x7f587c0ad660 2024-03-28T02:09:32.317 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:32.319 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:32.325 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:32.326 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:32.326 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:32.326 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:32.329 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:32.330 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:32.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.338+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 --> [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5890062d10 con 0x7f587c0ad660 2024-03-28T02:09:32.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.341+0000 7f58867fc700 1 -- 172.21.15.106:0/445927101 <== osd.10 v2:172.21.15.153:6824/1470813442 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5890062d10 con 0x7f587c0ad660 2024-03-28T02:09:32.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.347+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 >> [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] conn(0x7f587c0ad660 msgr2=0x7f587c0afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:32.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.347+0000 7f58978cb700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] conn(0x7f587c0ad660 0x7f587c0afaa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:32.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.348+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f587c0a29a0 msgr2=0x7f587c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:32.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.348+0000 7f58978cb700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f587c0a29a0 0x7f587c0a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f588c009fd0 tx=0x7f588c009260 comp rx=0 tx=0).stop 2024-03-28T02:09:32.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.348+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5890107120 msgr2=0x7f58901a4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:32.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.348+0000 7f58978cb700 1 --2- 172.21.15.106:0/445927101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5890107120 0x7f58901a4730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f588800ef30 tx=0x7f588800c3a0 comp rx=0 tx=0).stop 2024-03-28T02:09:32.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.349+0000 7f5895e68700 1 -- 172.21.15.106:0/445927101 reap_dead start 2024-03-28T02:09:32.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.349+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 shutdown_connections 2024-03-28T02:09:32.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.349+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 >> 172.21.15.106:0/445927101 conn(0x7f58900780a0 msgr2=0x7f58900ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:32.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.352+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 shutdown_connections 2024-03-28T02:09:32.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:32.353+0000 7f58978cb700 1 -- 172.21.15.106:0/445927101 wait complete. 2024-03-28T02:09:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.218+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3148218480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26d806b2a0 msgr2=0x7f26d810efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.218+0000 7f26dfa1c700 1 --2- 172.21.15.106:0/3148218480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26d806b2a0 0x7f26d810efe0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b0d0 tx=0x7f26d000b3e0 comp rx=0 tx=0).stop 2024-03-28T02:09:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.218+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3148218480 shutdown_connections 2024-03-28T02:09:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.218+0000 7f26dfa1c700 1 --2- 172.21.15.106:0/3148218480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26d806b2a0 0x7f26d810efe0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.218+0000 7f26dfa1c700 1 --2- 172.21.15.106:0/3148218480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26d81081c0 0x7f26d81085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.218+0000 7f26dfa1c700 1 --2- 172.21.15.106:0/3148218480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d806a8e0 0x7f26d806ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.218+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3148218480 >> 172.21.15.106:0/3148218480 conn(0x7f26d80780a0 msgr2=0x7f26d80784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.218+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3148218480 shutdown_connections 2024-03-28T02:09:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.218+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3148218480 wait complete. 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dfa1c700 1 Processor -- start 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dfa1c700 1 -- start start 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dfa1c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d806a8e0 0x7f26d819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dfa1c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26d806b2a0 0x7f26d819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dfa1c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26d81081c0 0x7f26d8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dfa1c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d8112460 con 0x7f26d806b2a0 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dfa1c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d81122e0 con 0x7f26d806a8e0 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dfa1c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d81125e0 con 0x7f26d81081c0 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dcfb7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26d806b2a0 0x7f26d819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:33.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dcfb7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26d806b2a0 0x7f26d819e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52300/0 (socket says 172.21.15.106:52300) 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dd7b8700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d806a8e0 0x7f26d819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26dcfb7700 1 -- 172.21.15.106:0/3980347863 learned_addr learned my addr 172.21.15.106:0/3980347863 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.219+0000 7f26ddfb9700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26d81081c0 0x7f26d8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26dcfb7700 1 -- 172.21.15.106:0/3980347863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26d81081c0 msgr2=0x7f26d8198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26dcfb7700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26d81081c0 0x7f26d8198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26dcfb7700 1 -- 172.21.15.106:0/3980347863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d806a8e0 msgr2=0x7f26d819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26dcfb7700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d806a8e0 0x7f26d819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26dcfb7700 1 -- 172.21.15.106:0/3980347863 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d8198d80 con 0x7f26d806b2a0 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26ddfb9700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26d81081c0 0x7f26d8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26dcfb7700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26d806b2a0 0x7f26d819e9a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f26c800e9a0 tx=0x7f26c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26ce7fc700 1 -- 172.21.15.106:0/3980347863 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c800ca10 con 0x7f26d806b2a0 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26ce7fc700 1 -- 172.21.15.106:0/3980347863 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26c800cbb0 con 0x7f26d806b2a0 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d8199070 con 0x7f26d806b2a0 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26ce7fc700 1 -- 172.21.15.106:0/3980347863 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8018680 con 0x7f26d806b2a0 2024-03-28T02:09:33.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.220+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d81a5330 con 0x7f26d806b2a0 2024-03-28T02:09:33.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.221+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f26d8100550 con 0x7f26d806b2a0 2024-03-28T02:09:33.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.222+0000 7f26ce7fc700 1 -- 172.21.15.106:0/3980347863 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f26c8010430 con 0x7f26d806b2a0 2024-03-28T02:09:33.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.223+0000 7f26ce7fc700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26c40a2920 0x7f26c40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:33.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.223+0000 7f26dd7b8700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26c40a2920 0x7f26c40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:33.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.223+0000 7f26ce7fc700 1 -- 172.21.15.106:0/3980347863 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f26c80cf880 con 0x7f26d806b2a0 2024-03-28T02:09:33.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.223+0000 7f26ce7fc700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] conn(0x7f26c40ad5e0 0x7f26c40afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:33.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.223+0000 7f26ce7fc700 1 -- 172.21.15.106:0/3980347863 --> [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26c801bec0 con 0x7f26c40ad5e0 2024-03-28T02:09:33.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.223+0000 7f26ce7fc700 1 -- 172.21.15.106:0/3980347863 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26c80cfcd0 con 0x7f26d806b2a0 2024-03-28T02:09:33.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.223+0000 7f26ddfb9700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] conn(0x7f26c40ad5e0 0x7f26c40afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:33.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.223+0000 7f26dd7b8700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26c40a2920 0x7f26c40a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009c60 tx=0x7f26d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:33.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.224+0000 7f26ddfb9700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] conn(0x7f26c40ad5e0 0x7f26c40afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:33.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.224+0000 7f26ce7fc700 1 -- 172.21.15.106:0/3980347863 <== osd.0 v2:172.21.15.106:6802/1092015924 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f26c801bec0 con 0x7f26c40ad5e0 2024-03-28T02:09:33.239 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.246+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 --> [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f26d8103870 con 0x7f26c40ad5e0 2024-03-28T02:09:33.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.246+0000 7f26ce7fc700 1 -- 172.21.15.106:0/3980347863 <== osd.0 v2:172.21.15.106:6802/1092015924 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f26d8103870 con 0x7f26c40ad5e0 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] conn(0x7f26c40ad5e0 msgr2=0x7f26c40afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] conn(0x7f26c40ad5e0 0x7f26c40afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26c40a2920 msgr2=0x7f26c40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26c40a2920 0x7f26c40a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009c60 tx=0x7f26d4009260 comp rx=0 tx=0).stop 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26d806b2a0 msgr2=0x7f26d819e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 --2- 172.21.15.106:0/3980347863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26d806b2a0 0x7f26d819e9a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f26c800e9a0 tx=0x7f26c800ecb0 comp rx=0 tx=0).stop 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26ddfb9700 1 -- 172.21.15.106:0/3980347863 reap_dead start 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 shutdown_connections 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 >> 172.21.15.106:0/3980347863 conn(0x7f26d80780a0 msgr2=0x7f26d80ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 shutdown_connections 2024-03-28T02:09:33.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:33.247+0000 7f26dfa1c700 1 -- 172.21.15.106:0/3980347863 wait complete. 2024-03-28T02:09:33.504 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:33 smithi106 ceph-mon[31965]: pgmap v314: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-28T02:09:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:33 smithi153 ceph-mon[36228]: pgmap v314: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-28T02:09:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:33 smithi178 ceph-mon[36521]: pgmap v314: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-28T02:09:34.416 INFO:teuthology.orchestra.run.smithi106.stdout:309237645339 2024-03-28T02:09:34.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T02:09:35.470 INFO:teuthology.orchestra.run.smithi106.stdout:34359738446 2024-03-28T02:09:35.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T02:09:35.496 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:35 smithi106 ceph-mon[31965]: pgmap v315: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T02:09:35.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:35 smithi153 ceph-mon[36228]: pgmap v315: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T02:09:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:35 smithi178 ceph-mon[36521]: pgmap v315: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T02:09:36.615 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.697+0000 7f9b6c53f700 1 -- 172.21.15.106:0/35271173 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b64102e20 msgr2=0x7f9b64106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.697+0000 7f9b6c53f700 1 --2- 172.21.15.106:0/35271173 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b64102e20 0x7f9b64106dc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009990 tx=0x7f9b54009ca0 comp rx=0 tx=0).stop 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.697+0000 7f9b6c53f700 1 -- 172.21.15.106:0/35271173 shutdown_connections 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.697+0000 7f9b6c53f700 1 --2- 172.21.15.106:0/35271173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b641074a0 0x7f9b6410f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.697+0000 7f9b6c53f700 1 --2- 172.21.15.106:0/35271173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b64102470 0x7f9b64102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.697+0000 7f9b6c53f700 1 --2- 172.21.15.106:0/35271173 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b64102e20 0x7f9b64106dc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.697+0000 7f9b6c53f700 1 -- 172.21.15.106:0/35271173 >> 172.21.15.106:0/35271173 conn(0x7f9b64079220 msgr2=0x7f9b64079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.697+0000 7f9b6c53f700 1 -- 172.21.15.106:0/35271173 shutdown_connections 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.697+0000 7f9b6c53f700 1 -- 172.21.15.106:0/35271173 wait complete. 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6c53f700 1 Processor -- start 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6c53f700 1 -- start start 2024-03-28T02:09:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6c53f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b64102470 0x7f9b64074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6c53f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b64102e20 0x7f9b640749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6c53f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b641074a0 0x7f9b64074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6c53f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b64112440 con 0x7f9b64102e20 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6c53f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b641122c0 con 0x7f9b64102470 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6c53f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b641125c0 con 0x7f9b641074a0 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b69ada700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b64102e20 0x7f9b640749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b69ada700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b64102e20 0x7f9b640749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52320/0 (socket says 172.21.15.106:52320) 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b69ada700 1 -- 172.21.15.106:0/3727302565 learned_addr learned my addr 172.21.15.106:0/3727302565 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6a2db700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b64102470 0x7f9b64074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.698+0000 7f9b6aadc700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b641074a0 0x7f9b64074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b69ada700 1 -- 172.21.15.106:0/3727302565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b641074a0 msgr2=0x7f9b64074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b69ada700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b641074a0 0x7f9b64074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b69ada700 1 -- 172.21.15.106:0/3727302565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b64102470 msgr2=0x7f9b64074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:36.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b69ada700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b64102470 0x7f9b64074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:36.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b69ada700 1 -- 172.21.15.106:0/3727302565 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b641a6e40 con 0x7f9b64102e20 2024-03-28T02:09:36.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b6a2db700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b64102470 0x7f9b64074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:36.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b6aadc700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b641074a0 0x7f9b64074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:36.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b69ada700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b64102e20 0x7f9b640749d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9b54000c00 tx=0x7f9b5400bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:36.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b5b7fe700 1 -- 172.21.15.106:0/3727302565 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54027070 con 0x7f9b64102e20 2024-03-28T02:09:36.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b641a70d0 con 0x7f9b64102e20 2024-03-28T02:09:36.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b5b7fe700 1 -- 172.21.15.106:0/3727302565 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b54003d70 con 0x7f9b64102e20 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b641a7580 con 0x7f9b64102e20 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.699+0000 7f9b5b7fe700 1 -- 172.21.15.106:0/3727302565 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5400f460 con 0x7f9b64102e20 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.700+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9b48001030 con 0x7f9b64102e20 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.701+0000 7f9b5b7fe700 1 -- 172.21.15.106:0/3727302565 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9b5402a070 con 0x7f9b64102e20 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.702+0000 7f9b5b7fe700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b500a2920 0x7f9b500a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.702+0000 7f9b6a2db700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b500a2920 0x7f9b500a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.702+0000 7f9b5b7fe700 1 -- 172.21.15.106:0/3727302565 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9b540d0180 con 0x7f9b64102e20 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.702+0000 7f9b5b7fe700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] conn(0x7f9b500ad5e0 0x7f9b500afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.702+0000 7f9b5b7fe700 1 -- 172.21.15.106:0/3727302565 --> [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b540043d0 con 0x7f9b500ad5e0 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.702+0000 7f9b5b7fe700 1 -- 172.21.15.106:0/3727302565 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b540d0530 con 0x7f9b64102e20 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.702+0000 7f9b6a2db700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b500a2920 0x7f9b500a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9b60011490 tx=0x7f9b60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.702+0000 7f9b6aadc700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] conn(0x7f9b500ad5e0 0x7f9b500afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.703+0000 7f9b6aadc700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] conn(0x7f9b500ad5e0 0x7f9b500afa20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.704+0000 7f9b5b7fe700 1 -- 172.21.15.106:0/3727302565 <== osd.4 v2:172.21.15.153:6808/3126868610 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b540043d0 con 0x7f9b500ad5e0 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.722+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 --> [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9b48002e60 con 0x7f9b500ad5e0 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b5b7fe700 1 -- 172.21.15.106:0/3727302565 <== osd.4 v2:172.21.15.153:6808/3126868610 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9b48002e60 con 0x7f9b500ad5e0 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 >> [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] conn(0x7f9b500ad5e0 msgr2=0x7f9b500afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] conn(0x7f9b500ad5e0 0x7f9b500afa20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b500a2920 msgr2=0x7f9b500a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9b500a2920 0x7f9b500a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9b60011490 tx=0x7f9b60009260 comp rx=0 tx=0).stop 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b64102e20 msgr2=0x7f9b640749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 --2- 172.21.15.106:0/3727302565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b64102e20 0x7f9b640749d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9b54000c00 tx=0x7f9b5400bab0 comp rx=0 tx=0).stop 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6aadc700 1 -- 172.21.15.106:0/3727302565 reap_dead start 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 shutdown_connections 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 >> 172.21.15.106:0/3727302565 conn(0x7f9b64079220 msgr2=0x7f9b641019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 shutdown_connections 2024-03-28T02:09:36.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:35.723+0000 7f9b6c53f700 1 -- 172.21.15.106:0/3727302565 wait complete. 2024-03-28T02:09:36.823 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:37.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:37 smithi106 ceph-mon[31965]: pgmap v316: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T02:09:37.514 INFO:teuthology.orchestra.run.smithi106.stdout:141733920827 2024-03-28T02:09:37.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T02:09:37.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:37 smithi153 ceph-mon[36228]: pgmap v316: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T02:09:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:37 smithi178 ceph-mon[36521]: pgmap v316: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T02:09:39.722 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:39 smithi106 ceph-mon[31965]: pgmap v317: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T02:09:39.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:39 smithi153 ceph-mon[36228]: pgmap v317: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T02:09:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:39 smithi178 ceph-mon[36521]: pgmap v317: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T02:09:41.709 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:41 smithi106 ceph-mon[31965]: pgmap v318: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T02:09:41.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:41 smithi153 ceph-mon[36228]: pgmap v318: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T02:09:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:41 smithi178 ceph-mon[36521]: pgmap v318: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T02:09:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-28T02:09:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-28T02:09:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-28T02:09:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-03-28T02:09:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]: dispatch 2024-03-28T02:09:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-03-28T02:09:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-28T02:09:42.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-28T02:09:42.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]}]: dispatch 2024-03-28T02:09:42.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-28T02:09:42.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-28T02:09:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-28T02:09:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-03-28T02:09:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]: dispatch 2024-03-28T02:09:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-03-28T02:09:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-28T02:09:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-28T02:09:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]}]: dispatch 2024-03-28T02:09:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-28T02:09:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-28T02:09:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-28T02:09:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-03-28T02:09:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]: dispatch 2024-03-28T02:09:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-03-28T02:09:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-28T02:09:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-28T02:09:42.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]}]: dispatch 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: pgmap v319: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]': finished 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]}]': finished 2024-03-28T02:09:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:43 smithi153 ceph-mon[36228]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T02:09:43.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: pgmap v319: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T02:09:43.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-28T02:09:43.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-28T02:09:43.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-28T02:09:43.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-03-28T02:09:43.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]': finished 2024-03-28T02:09:43.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-03-28T02:09:43.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-28T02:09:43.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-28T02:09:43.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]}]': finished 2024-03-28T02:09:43.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:43 smithi106 ceph-mon[31965]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: pgmap v319: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]': finished 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]}]': finished 2024-03-28T02:09:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:43 smithi178 ceph-mon[36521]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T02:09:44.665 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:44 smithi106 ceph-mon[31965]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T02:09:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.674+0000 7f83accfc700 1 -- 172.21.15.106:0/2976314568 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83a81074f0 msgr2=0x7f83a810f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.674+0000 7f83accfc700 1 --2- 172.21.15.106:0/2976314568 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83a81074f0 0x7f83a810f0c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f83a000b0d0 tx=0x7f83a000b3e0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.675+0000 7f83accfc700 1 -- 172.21.15.106:0/2976314568 shutdown_connections 2024-03-28T02:09:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.675+0000 7f83accfc700 1 --2- 172.21.15.106:0/2976314568 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83a81074f0 0x7f83a810f0c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.675+0000 7f83accfc700 1 --2- 172.21.15.106:0/2976314568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83a8102410 0x7f83a81027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.675+0000 7f83accfc700 1 --2- 172.21.15.106:0/2976314568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83a8102dc0 0x7f83a8106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.675+0000 7f83accfc700 1 -- 172.21.15.106:0/2976314568 >> 172.21.15.106:0/2976314568 conn(0x7f83a80fdc60 msgr2=0x7f83a8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.675+0000 7f83accfc700 1 -- 172.21.15.106:0/2976314568 shutdown_connections 2024-03-28T02:09:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.675+0000 7f83accfc700 1 -- 172.21.15.106:0/2976314568 wait complete. 2024-03-28T02:09:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.675+0000 7f83accfc700 1 Processor -- start 2024-03-28T02:09:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83accfc700 1 -- start start 2024-03-28T02:09:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83accfc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83a8102410 0x7f83a819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83accfc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83a8102dc0 0x7f83a819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83accfc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83a81074f0 0x7f83a8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83accfc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a81124e0 con 0x7f83a81074f0 2024-03-28T02:09:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83accfc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a8112360 con 0x7f83a8102410 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83accfc700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a8112660 con 0x7f83a8102dc0 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a6d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83a81074f0 0x7f83a8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a6d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83a81074f0 0x7f83a8198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:44964/0 (socket says 172.21.15.106:44964) 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a6d9d700 1 -- 172.21.15.106:0/1888657818 learned_addr learned my addr 172.21.15.106:0/1888657818 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a659c700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83a8102410 0x7f83a819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a5d9b700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83a8102dc0 0x7f83a819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a659c700 1 -- 172.21.15.106:0/1888657818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83a8102dc0 msgr2=0x7f83a819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a659c700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83a8102dc0 0x7f83a819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a659c700 1 -- 172.21.15.106:0/1888657818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83a81074f0 msgr2=0x7f83a8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a659c700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83a81074f0 0x7f83a8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a659c700 1 -- 172.21.15.106:0/1888657818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a8199000 con 0x7f83a8102410 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a6d9d700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83a81074f0 0x7f83a8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.676+0000 7f83a659c700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83a8102410 0x7f83a819e650 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f839800e9f0 tx=0x7f839800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.677+0000 7f83977fe700 1 -- 172.21.15.106:0/1888657818 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839800c9c0 con 0x7f83a8102410 2024-03-28T02:09:44.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.677+0000 7f83977fe700 1 -- 172.21.15.106:0/1888657818 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f839800cb60 con 0x7f83a8102410 2024-03-28T02:09:44.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.677+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a81992f0 con 0x7f83a8102410 2024-03-28T02:09:44.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.677+0000 7f83977fe700 1 -- 172.21.15.106:0/1888657818 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398018680 con 0x7f83a8102410 2024-03-28T02:09:44.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.677+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a81a55d0 con 0x7f83a8102410 2024-03-28T02:09:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.677+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f83a80628e0 con 0x7f83a8102410 2024-03-28T02:09:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.679+0000 7f83977fe700 1 -- 172.21.15.106:0/1888657818 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8398010430 con 0x7f83a8102410 2024-03-28T02:09:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.679+0000 7f83977fe700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f83900a2920 0x7f83900a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.679+0000 7f83a5d9b700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f83900a2920 0x7f83900a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.679+0000 7f83977fe700 1 -- 172.21.15.106:0/1888657818 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f83980cf550 con 0x7f83a8102410 2024-03-28T02:09:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.679+0000 7f83977fe700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] conn(0x7f83900ada00 0x7f83900afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.679+0000 7f83977fe700 1 -- 172.21.15.106:0/1888657818 --> [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f839801bec0 con 0x7f83900ada00 2024-03-28T02:09:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.679+0000 7f83977fe700 1 -- 172.21.15.106:0/1888657818 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83980cf900 con 0x7f83a8102410 2024-03-28T02:09:44.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.680+0000 7f83a6d9d700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] conn(0x7f83900ada00 0x7f83900afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.680+0000 7f83a5d9b700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f83900a2920 0x7f83900a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f83a8199ee0 tx=0x7f839c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:44.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.680+0000 7f83a6d9d700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] conn(0x7f83900ada00 0x7f83900afea0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:44.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.681+0000 7f83977fe700 1 -- 172.21.15.106:0/1888657818 <== osd.5 v2:172.21.15.178:6808/3071904021 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f839801bec0 con 0x7f83900ada00 2024-03-28T02:09:44.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 --> [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f83a8062d60 con 0x7f83900ada00 2024-03-28T02:09:44.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83977fe700 1 -- 172.21.15.106:0/1888657818 <== osd.5 v2:172.21.15.178:6808/3071904021 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f83a8062d60 con 0x7f83900ada00 2024-03-28T02:09:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 >> [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] conn(0x7f83900ada00 msgr2=0x7f83900afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83accfc700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] conn(0x7f83900ada00 0x7f83900afea0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f83900a2920 msgr2=0x7f83900a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83accfc700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f83900a2920 0x7f83900a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f83a8199ee0 tx=0x7f839c0073d0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83a8102410 msgr2=0x7f83a819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83accfc700 1 --2- 172.21.15.106:0/1888657818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83a8102410 0x7f83a819e650 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f839800e9f0 tx=0x7f839800ed00 comp rx=0 tx=0).stop 2024-03-28T02:09:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83a6d9d700 1 -- 172.21.15.106:0/1888657818 reap_dead start 2024-03-28T02:09:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 shutdown_connections 2024-03-28T02:09:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.697+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 >> 172.21.15.106:0/1888657818 conn(0x7f83a80fdc60 msgr2=0x7f83a80fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:44.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.698+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 shutdown_connections 2024-03-28T02:09:44.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.698+0000 7f83accfc700 1 -- 172.21.15.106:0/1888657818 wait complete. 2024-03-28T02:09:44.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:44 smithi153 ceph-mon[36228]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T02:09:44.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:44 smithi178 ceph-mon[36521]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T02:09:44.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.890+0000 7f3a60a18700 1 -- 172.21.15.106:0/715403503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a5c1070d0 msgr2=0x7f3a5c1074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.890+0000 7f3a60a18700 1 --2- 172.21.15.106:0/715403503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a5c1070d0 0x7f3a5c1074b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009990 tx=0x7f3a50009ca0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.890+0000 7f3a60a18700 1 -- 172.21.15.106:0/715403503 shutdown_connections 2024-03-28T02:09:44.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.890+0000 7f3a60a18700 1 --2- 172.21.15.106:0/715403503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c101a30 0x7f3a5c10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.890+0000 7f3a60a18700 1 --2- 172.21.15.106:0/715403503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a5c1070d0 0x7f3a5c1074b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.890+0000 7f3a60a18700 1 --2- 172.21.15.106:0/715403503 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c101090 0x7f3a5c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.890+0000 7f3a60a18700 1 -- 172.21.15.106:0/715403503 >> 172.21.15.106:0/715403503 conn(0x7f3a5c077fc0 msgr2=0x7f3a5c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:44.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.890+0000 7f3a60a18700 1 -- 172.21.15.106:0/715403503 shutdown_connections 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.890+0000 7f3a60a18700 1 -- 172.21.15.106:0/715403503 wait complete. 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a60a18700 1 Processor -- start 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a60a18700 1 -- start start 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a60a18700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c101090 0x7f3a5c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a60a18700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c101a30 0x7f3a5c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a60a18700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a5c1070d0 0x7f3a5c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a60a18700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c111310 con 0x7f3a5c1070d0 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a60a18700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c111190 con 0x7f3a5c101090 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a60a18700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c111490 con 0x7f3a5c101a30 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a5ad9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a5c1070d0 0x7f3a5c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a5ad9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a5c1070d0 0x7f3a5c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:44986/0 (socket says 172.21.15.106:44986) 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a5a59c700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c101090 0x7f3a5c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a5ad9d700 1 -- 172.21.15.106:0/2350492075 learned_addr learned my addr 172.21.15.106:0/2350492075 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.891+0000 7f3a59d9b700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c101a30 0x7f3a5c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a5ad9d700 1 -- 172.21.15.106:0/2350492075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c101a30 msgr2=0x7f3a5c19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a5ad9d700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c101a30 0x7f3a5c19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a5ad9d700 1 -- 172.21.15.106:0/2350492075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c101090 msgr2=0x7f3a5c19e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a5ad9d700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c101090 0x7f3a5c19e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a5ad9d700 1 -- 172.21.15.106:0/2350492075 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c198f90 con 0x7f3a5c1070d0 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a5a59c700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c101090 0x7f3a5c19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a5ad9d700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a5c1070d0 0x7f3a5c1986b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400ef30 tx=0x7f3a5400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a4b7fe700 1 -- 172.21.15.106:0/2350492075 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540092e0 con 0x7f3a5c1070d0 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5c199220 con 0x7f3a5c1070d0 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a4b7fe700 1 -- 172.21.15.106:0/2350492075 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a54004bb0 con 0x7f3a5c1070d0 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a4b7fe700 1 -- 172.21.15.106:0/2350492075 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a54004280 con 0x7f3a5c1070d0 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.892+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5c1a5560 con 0x7f3a5c1070d0 2024-03-28T02:09:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.893+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a5c06a150 con 0x7f3a5c1070d0 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.894+0000 7f3a4b7fe700 1 -- 172.21.15.106:0/2350492075 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f3a5401b030 con 0x7f3a5c1070d0 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.894+0000 7f3a4b7fe700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3a440a2920 0x7f3a440a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.895+0000 7f3a5a59c700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3a440a2920 0x7f3a440a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.895+0000 7f3a4b7fe700 1 -- 172.21.15.106:0/2350492075 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f3a540cf2b0 con 0x7f3a5c1070d0 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.895+0000 7f3a4b7fe700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] conn(0x7f3a440ada00 0x7f3a440afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.895+0000 7f3a4b7fe700 1 -- 172.21.15.106:0/2350492075 --> [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a5401fe50 con 0x7f3a440ada00 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.895+0000 7f3a4b7fe700 1 -- 172.21.15.106:0/2350492075 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a540cf700 con 0x7f3a5c1070d0 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.895+0000 7f3a5a59c700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3a440a2920 0x7f3a440a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000b2e0 tx=0x7f3a50005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.895+0000 7f3a59d9b700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] conn(0x7f3a440ada00 0x7f3a440afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.895+0000 7f3a59d9b700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] conn(0x7f3a440ada00 0x7f3a440afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.896+0000 7f3a4b7fe700 1 -- 172.21.15.106:0/2350492075 <== osd.1 v2:172.21.15.153:6800/2006532693 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a5401fe50 con 0x7f3a440ada00 2024-03-28T02:09:44.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.913+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 --> [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a5c102780 con 0x7f3a440ada00 2024-03-28T02:09:44.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.913+0000 7f3a4b7fe700 1 -- 172.21.15.106:0/2350492075 <== osd.1 v2:172.21.15.153:6800/2006532693 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3a5c102780 con 0x7f3a440ada00 2024-03-28T02:09:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.913+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 >> [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] conn(0x7f3a440ada00 msgr2=0x7f3a440afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.913+0000 7f3a60a18700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] conn(0x7f3a440ada00 0x7f3a440afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.913+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3a440a2920 msgr2=0x7f3a440a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.913+0000 7f3a60a18700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f3a440a2920 0x7f3a440a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000b2e0 tx=0x7f3a50005960 comp rx=0 tx=0).stop 2024-03-28T02:09:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.913+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a5c1070d0 msgr2=0x7f3a5c1986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.913+0000 7f3a60a18700 1 --2- 172.21.15.106:0/2350492075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a5c1070d0 0x7f3a5c1986b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400ef30 tx=0x7f3a5400c3a0 comp rx=0 tx=0).stop 2024-03-28T02:09:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.914+0000 7f3a5ad9d700 1 -- 172.21.15.106:0/2350492075 reap_dead start 2024-03-28T02:09:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.914+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 shutdown_connections 2024-03-28T02:09:44.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.914+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 >> 172.21.15.106:0/2350492075 conn(0x7f3a5c077fc0 msgr2=0x7f3a5c1052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:44.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.914+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 shutdown_connections 2024-03-28T02:09:44.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:44.914+0000 7f3a60a18700 1 -- 172.21.15.106:0/2350492075 wait complete. 2024-03-28T02:09:45.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:45 smithi153 ceph-mon[36228]: pgmap v322: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T02:09:45.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:45 smithi106 ceph-mon[31965]: pgmap v322: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T02:09:45.771 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:45 smithi178 ceph-mon[36521]: pgmap v322: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T02:09:46.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.824+0000 7feb9333f700 1 -- 172.21.15.106:0/1493638279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb8c0743c0 msgr2=0x7feb8c0747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.824+0000 7feb9333f700 1 --2- 172.21.15.106:0/1493638279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb8c0743c0 0x7feb8c0747a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feb88009990 tx=0x7feb88009ca0 comp rx=0 tx=0).stop 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.824+0000 7feb9333f700 1 -- 172.21.15.106:0/1493638279 shutdown_connections 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.824+0000 7feb9333f700 1 --2- 172.21.15.106:0/1493638279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb8c10ebc0 0x7feb8c110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.824+0000 7feb9333f700 1 --2- 172.21.15.106:0/1493638279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb8c0743c0 0x7feb8c0747a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.824+0000 7feb9333f700 1 --2- 172.21.15.106:0/1493638279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb8c074ce0 0x7feb8c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.824+0000 7feb9333f700 1 -- 172.21.15.106:0/1493638279 >> 172.21.15.106:0/1493638279 conn(0x7feb8c079250 msgr2=0x7feb8c079660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.824+0000 7feb9333f700 1 -- 172.21.15.106:0/1493638279 shutdown_connections 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.824+0000 7feb9333f700 1 -- 172.21.15.106:0/1493638279 wait complete. 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb9333f700 1 Processor -- start 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb9333f700 1 -- start start 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb9333f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb8c0743c0 0x7feb8c1a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:46.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb9333f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb8c074ce0 0x7feb8c1a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb9333f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb8c10ebc0 0x7feb8c19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb9333f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c114680 con 0x7feb8c074ce0 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb9333f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c114500 con 0x7feb8c10ebc0 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb9333f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c114800 con 0x7feb8c0743c0 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb908da700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb8c074ce0 0x7feb8c1a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb910db700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb8c0743c0 0x7feb8c1a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb910db700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb8c0743c0 0x7feb8c1a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:55696/0 (socket says 172.21.15.106:55696) 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb910db700 1 -- 172.21.15.106:0/1053505476 learned_addr learned my addr 172.21.15.106:0/1053505476 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.825+0000 7feb918dc700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb8c10ebc0 0x7feb8c19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb908da700 1 -- 172.21.15.106:0/1053505476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb8c0743c0 msgr2=0x7feb8c1a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb908da700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb8c0743c0 0x7feb8c1a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb908da700 1 -- 172.21.15.106:0/1053505476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb8c10ebc0 msgr2=0x7feb8c19a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb908da700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb8c10ebc0 0x7feb8c19a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb908da700 1 -- 172.21.15.106:0/1053505476 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c19b0e0 con 0x7feb8c074ce0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb910db700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb8c0743c0 0x7feb8c1a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb918dc700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb8c10ebc0 0x7feb8c19a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb908da700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb8c074ce0 0x7feb8c1a0cb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00d750 tx=0x7feb7c00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb827fc700 1 -- 172.21.15.106:0/1053505476 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c014030 con 0x7feb8c074ce0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb827fc700 1 -- 172.21.15.106:0/1053505476 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb7c004500 con 0x7feb8c074ce0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c19b3d0 con 0x7feb8c074ce0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb827fc700 1 -- 172.21.15.106:0/1053505476 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c005420 con 0x7feb8c074ce0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.826+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c1a7640 con 0x7feb8c074ce0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.827+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb8c062920 con 0x7feb8c074ce0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.828+0000 7feb827fc700 1 -- 172.21.15.106:0/1053505476 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7feb7c004750 con 0x7feb8c074ce0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.828+0000 7feb827fc700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feb780a29a0 0x7feb780a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.829+0000 7feb910db700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feb780a29a0 0x7feb780a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:46.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.829+0000 7feb827fc700 1 -- 172.21.15.106:0/1053505476 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7feb7c0ceb00 con 0x7feb8c074ce0 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.829+0000 7feb827fc700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] conn(0x7feb780ada80 0x7feb780aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.829+0000 7feb827fc700 1 -- 172.21.15.106:0/1053505476 --> [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb7c01ae50 con 0x7feb780ada80 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.829+0000 7feb827fc700 1 -- 172.21.15.106:0/1053505476 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb7c0d32c0 con 0x7feb8c074ce0 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.829+0000 7feb918dc700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] conn(0x7feb780ada80 0x7feb780aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.829+0000 7feb910db700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feb780a29a0 0x7feb780a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb88009750 tx=0x7feb88005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.829+0000 7feb918dc700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] conn(0x7feb780ada80 0x7feb780aff20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.830+0000 7feb827fc700 1 -- 172.21.15.106:0/1053505476 <== osd.6 v2:172.21.15.106:6818/233665499 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7feb7c01ae50 con 0x7feb780ada80 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.847+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 --> [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feb8c062d20 con 0x7feb780ada80 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.847+0000 7feb827fc700 1 -- 172.21.15.106:0/1053505476 <== osd.6 v2:172.21.15.106:6818/233665499 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7feb8c062d20 con 0x7feb780ada80 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.847+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] conn(0x7feb780ada80 msgr2=0x7feb780aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:46.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb9333f700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] conn(0x7feb780ada80 0x7feb780aff20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:46.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feb780a29a0 msgr2=0x7feb780a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:46.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb9333f700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7feb780a29a0 0x7feb780a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb88009750 tx=0x7feb88005e80 comp rx=0 tx=0).stop 2024-03-28T02:09:46.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb8c074ce0 msgr2=0x7feb8c1a0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:46.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb9333f700 1 --2- 172.21.15.106:0/1053505476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb8c074ce0 0x7feb8c1a0cb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00d750 tx=0x7feb7c00da60 comp rx=0 tx=0).stop 2024-03-28T02:09:46.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb918dc700 1 -- 172.21.15.106:0/1053505476 reap_dead start 2024-03-28T02:09:46.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 shutdown_connections 2024-03-28T02:09:46.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 >> 172.21.15.106:0/1053505476 conn(0x7feb8c079250 msgr2=0x7feb8c0ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:46.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 shutdown_connections 2024-03-28T02:09:46.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:46.848+0000 7feb9333f700 1 -- 172.21.15.106:0/1053505476 wait complete. 2024-03-28T02:09:47.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.319+0000 7ffb5f447700 1 -- 172.21.15.106:0/1988310559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb581081c0 msgr2=0x7ffb581085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.319+0000 7ffb5f447700 1 --2- 172.21.15.106:0/1988310559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb581081c0 0x7ffb581085a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ffb540099c0 tx=0x7ffb54009cd0 comp rx=0 tx=0).stop 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.319+0000 7ffb5f447700 1 -- 172.21.15.106:0/1988310559 shutdown_connections 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.319+0000 7ffb5f447700 1 --2- 172.21.15.106:0/1988310559 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb58102ba0 0x7ffb5810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.319+0000 7ffb5f447700 1 --2- 172.21.15.106:0/1988310559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb581081c0 0x7ffb581085a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.319+0000 7ffb5f447700 1 --2- 172.21.15.106:0/1988310559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb581021e0 0x7ffb58102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.319+0000 7ffb5f447700 1 -- 172.21.15.106:0/1988310559 >> 172.21.15.106:0/1988310559 conn(0x7ffb580792c0 msgr2=0x7ffb580796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 -- 172.21.15.106:0/1988310559 shutdown_connections 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 -- 172.21.15.106:0/1988310559 wait complete. 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 Processor -- start 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 -- start start 2024-03-28T02:09:47.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb581021e0 0x7ffb5819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb58102ba0 0x7ffb5819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb581081c0 0x7ffb58198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb58112440 con 0x7ffb58102ba0 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb581122c0 con 0x7ffb581021e0 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5f447700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb581125c0 con 0x7ffb581081c0 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.320+0000 7ffb5d1e3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb581021e0 0x7ffb5819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5d1e3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb581021e0 0x7ffb5819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:59866/0 (socket says 172.21.15.106:59866) 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5d1e3700 1 -- 172.21.15.106:0/2674310963 learned_addr learned my addr 172.21.15.106:0/2674310963 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5d9e4700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb581081c0 0x7ffb58198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5c9e2700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb58102ba0 0x7ffb5819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5d1e3700 1 -- 172.21.15.106:0/2674310963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb581081c0 msgr2=0x7ffb58198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5d1e3700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb581081c0 0x7ffb58198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5d1e3700 1 -- 172.21.15.106:0/2674310963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb58102ba0 msgr2=0x7ffb5819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5d1e3700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb58102ba0 0x7ffb5819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5d1e3700 1 -- 172.21.15.106:0/2674310963 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb58199000 con 0x7ffb581021e0 2024-03-28T02:09:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5d1e3700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb581021e0 0x7ffb5819e650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb54006010 tx=0x7ffb5400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb4e7fc700 1 -- 172.21.15.106:0/2674310963 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54003680 con 0x7ffb581021e0 2024-03-28T02:09:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb58199290 con 0x7ffb581021e0 2024-03-28T02:09:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb4e7fc700 1 -- 172.21.15.106:0/2674310963 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb540050d0 con 0x7ffb581021e0 2024-03-28T02:09:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb4e7fc700 1 -- 172.21.15.106:0/2674310963 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54003db0 con 0x7ffb581021e0 2024-03-28T02:09:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.321+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb581a5400 con 0x7ffb581021e0 2024-03-28T02:09:47.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.322+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffb580628e0 con 0x7ffb581021e0 2024-03-28T02:09:47.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.323+0000 7ffb4e7fc700 1 -- 172.21.15.106:0/2674310963 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ffb54005280 con 0x7ffb581021e0 2024-03-28T02:09:47.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.324+0000 7ffb4e7fc700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffb440a2ac0 0x7ffb440a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:47.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.324+0000 7ffb5c9e2700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffb440a2ac0 0x7ffb440a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:47.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.324+0000 7ffb4e7fc700 1 -- 172.21.15.106:0/2674310963 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7ffb540cffe0 con 0x7ffb581021e0 2024-03-28T02:09:47.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.324+0000 7ffb4e7fc700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] conn(0x7ffb440adba0 0x7ffb440b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:47.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.324+0000 7ffb4e7fc700 1 -- 172.21.15.106:0/2674310963 --> [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb54003680 con 0x7ffb440adba0 2024-03-28T02:09:47.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.324+0000 7ffb4e7fc700 1 -- 172.21.15.106:0/2674310963 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb540d0410 con 0x7ffb581021e0 2024-03-28T02:09:47.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.324+0000 7ffb5d9e4700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] conn(0x7ffb440adba0 0x7ffb440b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:47.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.324+0000 7ffb5c9e2700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffb440a2ac0 0x7ffb440a4f80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ffb58199ee0 tx=0x7ffb48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:47.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.325+0000 7ffb5d9e4700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] conn(0x7ffb440adba0 0x7ffb440b0040 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:47.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.326+0000 7ffb4e7fc700 1 -- 172.21.15.106:0/2674310963 <== osd.8 v2:172.21.15.178:6816/3897935183 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffb54003680 con 0x7ffb440adba0 2024-03-28T02:09:47.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.342+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 --> [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffb58062d60 con 0x7ffb440adba0 2024-03-28T02:09:47.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb4e7fc700 1 -- 172.21.15.106:0/2674310963 <== osd.8 v2:172.21.15.178:6816/3897935183 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ffb58062d60 con 0x7ffb440adba0 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 >> [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] conn(0x7ffb440adba0 msgr2=0x7ffb440b0040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] conn(0x7ffb440adba0 0x7ffb440b0040 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffb440a2ac0 msgr2=0x7ffb440a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ffb440a2ac0 0x7ffb440a4f80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ffb58199ee0 tx=0x7ffb48009260 comp rx=0 tx=0).stop 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb581021e0 msgr2=0x7ffb5819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 --2- 172.21.15.106:0/2674310963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb581021e0 0x7ffb5819e650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb54006010 tx=0x7ffb5400f690 comp rx=0 tx=0).stop 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5d9e4700 1 -- 172.21.15.106:0/2674310963 reap_dead start 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 shutdown_connections 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 >> 172.21.15.106:0/2674310963 conn(0x7ffb580792c0 msgr2=0x7ffb58100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 shutdown_connections 2024-03-28T02:09:47.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:47.343+0000 7ffb5f447700 1 -- 172.21.15.106:0/2674310963 wait complete. 2024-03-28T02:09:47.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:47 smithi153 ceph-mon[36228]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s; 37 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T02:09:47.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:47 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:09:47.790 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:47 smithi106 ceph-mon[31965]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s; 37 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T02:09:47.790 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:47 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:09:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:47 smithi178 ceph-mon[36521]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s; 37 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T02:09:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:47 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:09:48.826 INFO:teuthology.orchestra.run.smithi106.stdout:197568495668 2024-03-28T02:09:48.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T02:09:49.494 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:49 smithi106 ceph-mon[31965]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 9 op/s; 37 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T02:09:49.494 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.494 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.494 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.494 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:49 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:49 smithi153 ceph-mon[36228]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 9 op/s; 37 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T02:09:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:49 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:49 smithi178 ceph-mon[36521]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 9 op/s; 37 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T02:09:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:49 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:09:50.150 INFO:teuthology.orchestra.run.smithi106.stdout:253403070506 2024-03-28T02:09:50.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T02:09:50.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.349+0000 7f0b4796c700 1 -- 172.21.15.106:0/2670700805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b40101a30 msgr2=0x7f0b4010df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:50.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.349+0000 7f0b4796c700 1 --2- 172.21.15.106:0/2670700805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b40101a30 0x7f0b4010df10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800b0a0 tx=0x7f0b3800b3b0 comp rx=0 tx=0).stop 2024-03-28T02:09:50.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.349+0000 7f0b4796c700 1 -- 172.21.15.106:0/2670700805 shutdown_connections 2024-03-28T02:09:50.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.349+0000 7f0b4796c700 1 --2- 172.21.15.106:0/2670700805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b40101a30 0x7f0b4010df10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:50.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.349+0000 7f0b4796c700 1 --2- 172.21.15.106:0/2670700805 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b401070d0 0x7f0b401074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:50.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.349+0000 7f0b4796c700 1 --2- 172.21.15.106:0/2670700805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b40101090 0x7f0b401014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:50.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.349+0000 7f0b4796c700 1 -- 172.21.15.106:0/2670700805 >> 172.21.15.106:0/2670700805 conn(0x7f0b40077fc0 msgr2=0x7f0b400783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:50.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.349+0000 7f0b4796c700 1 -- 172.21.15.106:0/2670700805 shutdown_connections 2024-03-28T02:09:50.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.349+0000 7f0b4796c700 1 -- 172.21.15.106:0/2670700805 wait complete. 2024-03-28T02:09:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b4796c700 1 Processor -- start 2024-03-28T02:09:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b4796c700 1 -- start start 2024-03-28T02:09:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b4796c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b40101090 0x7f0b401a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b4796c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b40101a30 0x7f0b401a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b4796c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b401070d0 0x7f0b4019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b4796c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b40111320 con 0x7f0b401070d0 2024-03-28T02:09:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b4796c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b401111a0 con 0x7f0b40101090 2024-03-28T02:09:50.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b4796c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b401114a0 con 0x7f0b40101a30 2024-03-28T02:09:50.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b45708700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b40101090 0x7f0b401a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:50.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b44f07700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b40101a30 0x7f0b401a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:50.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b44f07700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b40101a30 0x7f0b401a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:50306/0 (socket says 172.21.15.106:50306) 2024-03-28T02:09:50.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b44f07700 1 -- 172.21.15.106:0/3730928502 learned_addr learned my addr 172.21.15.106:0/3730928502 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:50.352 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b45f09700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b401070d0 0x7f0b4019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:50.352 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b44f07700 1 -- 172.21.15.106:0/3730928502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b40101090 msgr2=0x7f0b401a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:50.352 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b44f07700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b40101090 0x7f0b401a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:50.352 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b44f07700 1 -- 172.21.15.106:0/3730928502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b401070d0 msgr2=0x7f0b4019a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:50.352 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b44f07700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b401070d0 0x7f0b4019a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:50.352 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.350+0000 7f0b44f07700 1 -- 172.21.15.106:0/3730928502 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4019b060 con 0x7f0b40101a30 2024-03-28T02:09:50.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.351+0000 7f0b44f07700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b40101a30 0x7f0b401a0c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000e9a0 tx=0x7f0b3000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:50.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.351+0000 7f0b367fc700 1 -- 172.21.15.106:0/3730928502 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3000ca10 con 0x7f0b40101a30 2024-03-28T02:09:50.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.351+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4019b350 con 0x7f0b40101a30 2024-03-28T02:09:50.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.351+0000 7f0b367fc700 1 -- 172.21.15.106:0/3730928502 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b3000cbb0 con 0x7f0b40101a30 2024-03-28T02:09:50.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.351+0000 7f0b367fc700 1 -- 172.21.15.106:0/3730928502 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30018680 con 0x7f0b40101a30 2024-03-28T02:09:50.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.351+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b401a7610 con 0x7f0b40101a30 2024-03-28T02:09:50.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.352+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b4006a860 con 0x7f0b40101a30 2024-03-28T02:09:50.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.353+0000 7f0b367fc700 1 -- 172.21.15.106:0/3730928502 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0b30010430 con 0x7f0b40101a30 2024-03-28T02:09:50.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.354+0000 7f0b367fc700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0b2c0a2920 0x7f0b2c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:50.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.354+0000 7f0b367fc700 1 -- 172.21.15.106:0/3730928502 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f0b300cfce0 con 0x7f0b40101a30 2024-03-28T02:09:50.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.354+0000 7f0b45708700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0b2c0a2920 0x7f0b2c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:50.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.354+0000 7f0b367fc700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] conn(0x7f0b2c0ada00 0x7f0b2c0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:50.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.354+0000 7f0b367fc700 1 -- 172.21.15.106:0/3730928502 --> [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b2c0b0570 con 0x7f0b2c0ada00 2024-03-28T02:09:50.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.354+0000 7f0b367fc700 1 -- 172.21.15.106:0/3730928502 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b300d0090 con 0x7f0b40101a30 2024-03-28T02:09:50.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.354+0000 7f0b45f09700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] conn(0x7f0b2c0ada00 0x7f0b2c0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:50.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.354+0000 7f0b45708700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0b2c0a2920 0x7f0b2c0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0b4019ba40 tx=0x7f0b3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:50.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.355+0000 7f0b45f09700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] conn(0x7f0b2c0ada00 0x7f0b2c0afea0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:50.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.356+0000 7f0b367fc700 1 -- 172.21.15.106:0/3730928502 <== osd.11 v2:172.21.15.178:6824/3563501729 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0b2c0b0570 con 0x7f0b2c0ada00 2024-03-28T02:09:50.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.377+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 --> [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0b40102780 con 0x7f0b2c0ada00 2024-03-28T02:09:50.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b367fc700 1 -- 172.21.15.106:0/3730928502 <== osd.11 v2:172.21.15.178:6824/3563501729 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0b40102780 con 0x7f0b2c0ada00 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 >> [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] conn(0x7f0b2c0ada00 msgr2=0x7f0b2c0afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] conn(0x7f0b2c0ada00 0x7f0b2c0afea0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0b2c0a2920 msgr2=0x7f0b2c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f0b2c0a2920 0x7f0b2c0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0b4019ba40 tx=0x7f0b3c009260 comp rx=0 tx=0).stop 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b40101a30 msgr2=0x7f0b401a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 --2- 172.21.15.106:0/3730928502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b40101a30 0x7f0b401a0c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000e9a0 tx=0x7f0b3000ecb0 comp rx=0 tx=0).stop 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b45f09700 1 -- 172.21.15.106:0/3730928502 reap_dead start 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 shutdown_connections 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 >> 172.21.15.106:0/3730928502 conn(0x7f0b40077fc0 msgr2=0x7f0b40069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 shutdown_connections 2024-03-28T02:09:50.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:50.378+0000 7f0b4796c700 1 -- 172.21.15.106:0/3730928502 wait complete. 2024-03-28T02:09:51.264 INFO:teuthology.orchestra.run.smithi106.stdout:55834574924 2024-03-28T02:09:51.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T02:09:51.488 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:51 smithi106 ceph-mon[31965]: pgmap v325: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s; 37 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T02:09:51.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.697+0000 7fc2a2c72700 1 -- 172.21.15.106:0/1616145300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc29c101a30 msgr2=0x7fc29c10ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.697+0000 7fc2a2c72700 1 --2- 172.21.15.106:0/1616145300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc29c101a30 0x7fc29c10ded0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc29400b0a0 tx=0x7fc29400b3b0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.697+0000 7fc2a2c72700 1 -- 172.21.15.106:0/1616145300 shutdown_connections 2024-03-28T02:09:51.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.697+0000 7fc2a2c72700 1 --2- 172.21.15.106:0/1616145300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc29c101a30 0x7fc29c10ded0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.697+0000 7fc2a2c72700 1 --2- 172.21.15.106:0/1616145300 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc29c107120 0x7fc29c107500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.697+0000 7fc2a2c72700 1 --2- 172.21.15.106:0/1616145300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc29c101070 0x7fc29c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.697+0000 7fc2a2c72700 1 -- 172.21.15.106:0/1616145300 >> 172.21.15.106:0/1616145300 conn(0x7fc29c0780a0 msgr2=0x7fc29c0784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:51.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.698+0000 7fc2a2c72700 1 -- 172.21.15.106:0/1616145300 shutdown_connections 2024-03-28T02:09:51.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.698+0000 7fc2a2c72700 1 -- 172.21.15.106:0/1616145300 wait complete. 2024-03-28T02:09:51.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.698+0000 7fc2a2c72700 1 Processor -- start 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.698+0000 7fc2a2c72700 1 -- start start 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a2c72700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc29c101070 0x7fc29c1974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a2c72700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc29c101a30 0x7fc29c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a2c72700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc29c107120 0x7fc29c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a2c72700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c111350 con 0x7fc29c107120 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a2c72700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c1111d0 con 0x7fc29c101a30 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a2c72700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c1114d0 con 0x7fc29c101070 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a0a0e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc29c101070 0x7fc29c1974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc29bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc29c101a30 0x7fc29c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a0a0e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc29c101070 0x7fc29c1974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:50328/0 (socket says 172.21.15.106:50328) 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a0a0e700 1 -- 172.21.15.106:0/3607077403 learned_addr learned my addr 172.21.15.106:0/3607077403 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:51.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a120f700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc29c107120 0x7fc29c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a0a0e700 1 -- 172.21.15.106:0/3607077403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc29c101a30 msgr2=0x7fc29c197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a0a0e700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc29c101a30 0x7fc29c197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a0a0e700 1 -- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc29c107120 msgr2=0x7fc29c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a0a0e700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc29c107120 0x7fc29c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a0a0e700 1 -- 172.21.15.106:0/3607077403 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c1a6cf0 con 0x7fc29c101070 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc2a0a0e700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc29c101070 0x7fc29c1974e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e9f0 tx=0x7fc29000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.699+0000 7fc299ffb700 1 -- 172.21.15.106:0/3607077403 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000cbd0 con 0x7fc29c101070 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.700+0000 7fc299ffb700 1 -- 172.21.15.106:0/3607077403 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc29000ee90 con 0x7fc29c101070 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.700+0000 7fc299ffb700 1 -- 172.21.15.106:0/3607077403 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900056c0 con 0x7fc29c101070 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.700+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c1a6fe0 con 0x7fc29c101070 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.700+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c1a7430 con 0x7fc29c101070 2024-03-28T02:09:51.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.701+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc29c062920 con 0x7fc29c101070 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.702+0000 7fc299ffb700 1 -- 172.21.15.106:0/3607077403 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc290004750 con 0x7fc29c101070 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.702+0000 7fc299ffb700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc28c0a29a0 0x7fc28c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.702+0000 7fc29bfff700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc28c0a29a0 0x7fc28c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.702+0000 7fc299ffb700 1 -- 172.21.15.106:0/3607077403 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fc2900cf180 con 0x7fc29c101070 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.702+0000 7fc299ffb700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] conn(0x7fc28c0ada80 0x7fc28c0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.702+0000 7fc299ffb700 1 -- 172.21.15.106:0/3607077403 --> [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc29001ee50 con 0x7fc28c0ada80 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.702+0000 7fc299ffb700 1 -- 172.21.15.106:0/3607077403 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2900d6050 con 0x7fc29c101070 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.703+0000 7fc2a120f700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] conn(0x7fc28c0ada80 0x7fc28c0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.703+0000 7fc29bfff700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc28c0a29a0 0x7fc28c0a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc29c199640 tx=0x7fc288009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.703+0000 7fc2a120f700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] conn(0x7fc28c0ada80 0x7fc28c0aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:51.697 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.704+0000 7fc299ffb700 1 -- 172.21.15.106:0/3607077403 <== osd.9 v2:172.21.15.106:6826/4082459972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc29001ee50 con 0x7fc28c0ada80 2024-03-28T02:09:51.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.719+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 --> [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc29c062dd0 con 0x7fc28c0ada80 2024-03-28T02:09:51.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc299ffb700 1 -- 172.21.15.106:0/3607077403 <== osd.9 v2:172.21.15.106:6826/4082459972 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc29c062dd0 con 0x7fc28c0ada80 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] conn(0x7fc28c0ada80 msgr2=0x7fc28c0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] conn(0x7fc28c0ada80 0x7fc28c0aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc28c0a29a0 msgr2=0x7fc28c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc28c0a29a0 0x7fc28c0a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc29c199640 tx=0x7fc288009260 comp rx=0 tx=0).stop 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc29c101070 msgr2=0x7fc29c1974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 --2- 172.21.15.106:0/3607077403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc29c101070 0x7fc29c1974e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e9f0 tx=0x7fc29000ed00 comp rx=0 tx=0).stop 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a120f700 1 -- 172.21.15.106:0/3607077403 reap_dead start 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 shutdown_connections 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 >> 172.21.15.106:0/3607077403 conn(0x7fc29c0780a0 msgr2=0x7fc29c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 shutdown_connections 2024-03-28T02:09:51.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.720+0000 7fc2a2c72700 1 -- 172.21.15.106:0/3607077403 wait complete. 2024-03-28T02:09:51.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:51 smithi153 ceph-mon[36228]: pgmap v325: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s; 37 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T02:09:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:51 smithi178 ceph-mon[36521]: pgmap v325: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s; 37 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.899+0000 7faff724e700 1 -- 172.21.15.106:0/4021654285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faff0102df0 msgr2=0x7faff0106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.899+0000 7faff724e700 1 --2- 172.21.15.106:0/4021654285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faff0102df0 0x7faff0106e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009990 tx=0x7fafdc009ca0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.899+0000 7faff724e700 1 -- 172.21.15.106:0/4021654285 shutdown_connections 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.899+0000 7faff724e700 1 --2- 172.21.15.106:0/4021654285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff0107510 0x7faff010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.899+0000 7faff724e700 1 --2- 172.21.15.106:0/4021654285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faff0102440 0x7faff0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.899+0000 7faff724e700 1 --2- 172.21.15.106:0/4021654285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faff0102df0 0x7faff0106e30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.899+0000 7faff724e700 1 -- 172.21.15.106:0/4021654285 >> 172.21.15.106:0/4021654285 conn(0x7faff00792f0 msgr2=0x7faff0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.899+0000 7faff724e700 1 -- 172.21.15.106:0/4021654285 shutdown_connections 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.899+0000 7faff724e700 1 -- 172.21.15.106:0/4021654285 wait complete. 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff724e700 1 Processor -- start 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff724e700 1 -- start start 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff724e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faff0102440 0x7faff0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff724e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff0102df0 0x7faff00749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff724e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faff0107510 0x7faff0074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff724e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff01124f0 con 0x7faff0102df0 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff724e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff0112370 con 0x7faff0107510 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff724e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff0112670 con 0x7faff0102440 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7fafeffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff0102df0 0x7faff00749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff4fea700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faff0102440 0x7faff0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff57eb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faff0107510 0x7faff0074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff4fea700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faff0102440 0x7faff0074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:50344/0 (socket says 172.21.15.106:50344) 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7fafeffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff0102df0 0x7faff00749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39962/0 (socket says 172.21.15.106:39962) 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.900+0000 7faff4fea700 1 -- 172.21.15.106:0/1762296415 learned_addr learned my addr 172.21.15.106:0/1762296415 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7faff57eb700 1 -- 172.21.15.106:0/1762296415 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faff0102440 msgr2=0x7faff0074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7faff57eb700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faff0102440 0x7faff0074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7faff57eb700 1 -- 172.21.15.106:0/1762296415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff0102df0 msgr2=0x7faff00749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7faff57eb700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff0102df0 0x7faff00749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7faff57eb700 1 -- 172.21.15.106:0/1762296415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff01a6f40 con 0x7faff0107510 2024-03-28T02:09:51.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7faff57eb700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faff0107510 0x7faff0074f10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafe800efc0 tx=0x7fafe800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:51.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7fafedffb700 1 -- 172.21.15.106:0/1762296415 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe80093a0 con 0x7faff0107510 2024-03-28T02:09:51.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7fafedffb700 1 -- 172.21.15.106:0/1762296415 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafe800c940 con 0x7faff0107510 2024-03-28T02:09:51.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff01a71d0 con 0x7faff0107510 2024-03-28T02:09:51.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7fafedffb700 1 -- 172.21.15.106:0/1762296415 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8004550 con 0x7faff0107510 2024-03-28T02:09:51.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.901+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff01a76e0 con 0x7faff0107510 2024-03-28T02:09:51.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.902+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faff00628a0 con 0x7faff0107510 2024-03-28T02:09:51.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.903+0000 7fafedffb700 1 -- 172.21.15.106:0/1762296415 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fafe801b030 con 0x7faff0107510 2024-03-28T02:09:51.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.903+0000 7fafedffb700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafe00a2920 0x7fafe00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.904+0000 7faff4fea700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafe00a2920 0x7fafe00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.904+0000 7fafedffb700 1 -- 172.21.15.106:0/1762296415 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fafe80ce900 con 0x7faff0107510 2024-03-28T02:09:51.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.904+0000 7fafedffb700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] conn(0x7fafe00ada00 0x7fafe00afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:51.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.904+0000 7fafedffb700 1 -- 172.21.15.106:0/1762296415 --> [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafe801ae50 con 0x7fafe00ada00 2024-03-28T02:09:51.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.904+0000 7fafedffb700 1 -- 172.21.15.106:0/1762296415 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafe80cecb0 con 0x7faff0107510 2024-03-28T02:09:51.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.904+0000 7faff4fea700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafe00a2920 0x7fafe00a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009d60 tx=0x7fafe4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:51.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.904+0000 7fafeffff700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] conn(0x7fafe00ada00 0x7fafe00afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:51.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.904+0000 7fafeffff700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] conn(0x7fafe00ada00 0x7fafe00afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:51.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.905+0000 7fafedffb700 1 -- 172.21.15.106:0/1762296415 <== osd.7 v2:172.21.15.153:6816/1996966525 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fafe801ae50 con 0x7fafe00ada00 2024-03-28T02:09:51.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 --> [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faff0062ca0 con 0x7fafe00ada00 2024-03-28T02:09:51.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7fafedffb700 1 -- 172.21.15.106:0/1762296415 <== osd.7 v2:172.21.15.153:6816/1996966525 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7faff0062ca0 con 0x7fafe00ada00 2024-03-28T02:09:51.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 >> [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] conn(0x7fafe00ada00 msgr2=0x7fafe00afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff724e700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] conn(0x7fafe00ada00 0x7fafe00afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafe00a2920 msgr2=0x7fafe00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff724e700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fafe00a2920 0x7fafe00a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009d60 tx=0x7fafe4009260 comp rx=0 tx=0).stop 2024-03-28T02:09:51.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faff0107510 msgr2=0x7faff0074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:51.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff724e700 1 --2- 172.21.15.106:0/1762296415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faff0107510 0x7faff0074f10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafe800efc0 tx=0x7fafe800c3a0 comp rx=0 tx=0).stop 2024-03-28T02:09:51.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff57eb700 1 -- 172.21.15.106:0/1762296415 reap_dead start 2024-03-28T02:09:51.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 shutdown_connections 2024-03-28T02:09:51.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.922+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 >> 172.21.15.106:0/1762296415 conn(0x7faff00792f0 msgr2=0x7faff0100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:51.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.923+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 shutdown_connections 2024-03-28T02:09:51.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:51.923+0000 7faff724e700 1 -- 172.21.15.106:0/1762296415 wait complete. 2024-03-28T02:09:52.600 INFO:teuthology.orchestra.run.smithi106.stdout:287762808869 2024-03-28T02:09:52.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T02:09:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.144+0000 7f951a30b700 1 -- 172.21.15.106:0/2916712735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9514102ba0 msgr2=0x7f951410efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.144+0000 7f951a30b700 1 --2- 172.21.15.106:0/2916712735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9514102ba0 0x7f951410efe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f950c00b0d0 tx=0x7f950c00b3e0 comp rx=0 tx=0).stop 2024-03-28T02:09:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.144+0000 7f951a30b700 1 -- 172.21.15.106:0/2916712735 shutdown_connections 2024-03-28T02:09:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.144+0000 7f951a30b700 1 --2- 172.21.15.106:0/2916712735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9514102ba0 0x7f951410efe0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.144+0000 7f951a30b700 1 --2- 172.21.15.106:0/2916712735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95141081c0 0x7f95141085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.144+0000 7f951a30b700 1 --2- 172.21.15.106:0/2916712735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95141021e0 0x7f9514102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.144+0000 7f951a30b700 1 -- 172.21.15.106:0/2916712735 >> 172.21.15.106:0/2916712735 conn(0x7f95140792c0 msgr2=0x7f95140796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 -- 172.21.15.106:0/2916712735 shutdown_connections 2024-03-28T02:09:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 -- 172.21.15.106:0/2916712735 wait complete. 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 Processor -- start 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 -- start start 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95141021e0 0x7f951419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9514102ba0 0x7f951419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95141081c0 0x7f9514198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9514112470 con 0x7f9514102ba0 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141122f0 con 0x7f95141081c0 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.145+0000 7f951a30b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141125f0 con 0x7f95141021e0 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f9513fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95141021e0 0x7f951419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f95137fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9514102ba0 0x7f951419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f9513fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95141021e0 0x7f951419e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:50366/0 (socket says 172.21.15.106:50366) 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f95188a8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95141081c0 0x7f9514198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:53.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f9513fff700 1 -- 172.21.15.106:0/2189946843 learned_addr learned my addr 172.21.15.106:0/2189946843 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f9513fff700 1 -- 172.21.15.106:0/2189946843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95141081c0 msgr2=0x7f9514198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f9513fff700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95141081c0 0x7f9514198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f9513fff700 1 -- 172.21.15.106:0/2189946843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9514102ba0 msgr2=0x7f951419eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f9513fff700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9514102ba0 0x7f951419eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f9513fff700 1 -- 172.21.15.106:0/2189946843 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9514198f30 con 0x7f95141021e0 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f95188a8700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95141081c0 0x7f9514198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f9513fff700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95141021e0 0x7f951419e580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f950400e9a0 tx=0x7f950400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f95117fa700 1 -- 172.21.15.106:0/2189946843 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950400ca10 con 0x7f95141021e0 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9514199220 con 0x7f95141021e0 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f95117fa700 1 -- 172.21.15.106:0/2189946843 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f950400cbb0 con 0x7f95141021e0 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f95117fa700 1 -- 172.21.15.106:0/2189946843 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504018680 con 0x7f95141021e0 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.146+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95141a5500 con 0x7f95141021e0 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.147+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95140628a0 con 0x7f95141021e0 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.148+0000 7f95117fa700 1 -- 172.21.15.106:0/2189946843 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9504010430 con 0x7f95141021e0 2024-03-28T02:09:53.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.149+0000 7f95117fa700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f94fc0a2920 0x7f94fc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.149+0000 7f95137fe700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f94fc0a2920 0x7f94fc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.149+0000 7f95117fa700 1 -- 172.21.15.106:0/2189946843 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f95040cf9e0 con 0x7f95141021e0 2024-03-28T02:09:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.149+0000 7f95117fa700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] conn(0x7f94fc0ada00 0x7f94fc0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.149+0000 7f95117fa700 1 -- 172.21.15.106:0/2189946843 --> [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f950401bec0 con 0x7f94fc0ada00 2024-03-28T02:09:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.149+0000 7f95117fa700 1 -- 172.21.15.106:0/2189946843 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95040cfd90 con 0x7f95141021e0 2024-03-28T02:09:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.149+0000 7f95137fe700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f94fc0a2920 0x7f94fc0a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9514199e10 tx=0x7f9508009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.149+0000 7f95188a8700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] conn(0x7f94fc0ada00 0x7f94fc0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.150+0000 7f95188a8700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] conn(0x7f94fc0ada00 0x7f94fc0afea0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.150+0000 7f95117fa700 1 -- 172.21.15.106:0/2189946843 <== osd.2 v2:172.21.15.178:6800/3105645780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f950401bec0 con 0x7f94fc0ada00 2024-03-28T02:09:53.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.166+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 --> [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9514062cd0 con 0x7f94fc0ada00 2024-03-28T02:09:53.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f95117fa700 1 -- 172.21.15.106:0/2189946843 <== osd.2 v2:172.21.15.178:6800/3105645780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9514062cd0 con 0x7f94fc0ada00 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 >> [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] conn(0x7f94fc0ada00 msgr2=0x7f94fc0afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] conn(0x7f94fc0ada00 0x7f94fc0afea0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f94fc0a2920 msgr2=0x7f94fc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f94fc0a2920 0x7f94fc0a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9514199e10 tx=0x7f9508009260 comp rx=0 tx=0).stop 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95141021e0 msgr2=0x7f951419e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 --2- 172.21.15.106:0/2189946843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95141021e0 0x7f951419e580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f950400e9a0 tx=0x7f950400ecb0 comp rx=0 tx=0).stop 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f95188a8700 1 -- 172.21.15.106:0/2189946843 reap_dead start 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 shutdown_connections 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 >> 172.21.15.106:0/2189946843 conn(0x7f95140792c0 msgr2=0x7f9514100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 shutdown_connections 2024-03-28T02:09:53.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:53.167+0000 7f951a30b700 1 -- 172.21.15.106:0/2189946843 wait complete. 2024-03-28T02:09:53.608 INFO:teuthology.orchestra.run.smithi106.stdout:163208757304 2024-03-28T02:09:53.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T02:09:53.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:53 smithi153 ceph-mon[36228]: pgmap v326: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s; 29 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T02:09:53.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:53 smithi106 ceph-mon[31965]: pgmap v326: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s; 29 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T02:09:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:53 smithi178 ceph-mon[36521]: pgmap v326: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s; 29 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T02:09:54.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.032+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/4121615330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc100780 msgr2=0x7fa5bc100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:54.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.032+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/4121615330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc100780 0x7fa5bc100c00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009990 tx=0x7fa5a8009ca0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.032+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/4121615330 shutdown_connections 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.032+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/4121615330 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5bc101140 0x7fa5bc10f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.032+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/4121615330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc107200 0x7fa5bc1075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.032+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/4121615330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc100780 0x7fa5bc100c00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.032+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/4121615330 >> 172.21.15.106:0/4121615330 conn(0x7fa5bc077690 msgr2=0x7fa5bc0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.032+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/4121615330 shutdown_connections 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.032+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/4121615330 wait complete. 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c2dc3700 1 Processor -- start 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c2dc3700 1 -- start start 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c2dc3700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5bc100780 0x7fa5bc101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c2dc3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc101140 0x7fa5bc101ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:54.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c2dc3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc107200 0x7fa5bc1023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c2dc3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc1125e0 con 0x7fa5bc107200 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c2dc3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc112460 con 0x7fa5bc101140 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c2dc3700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc112760 con 0x7fa5bc100780 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c1360700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc107200 0x7fa5bc1023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c1360700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc107200 0x7fa5bc1023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:39994/0 (socket says 172.21.15.106:39994) 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c1360700 1 -- 172.21.15.106:0/2360945563 learned_addr learned my addr 172.21.15.106:0/2360945563 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c0b5f700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5bc100780 0x7fa5bc101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5bbfff700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc101140 0x7fa5bc101ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c1360700 1 -- 172.21.15.106:0/2360945563 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5bc100780 msgr2=0x7fa5bc101960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c1360700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5bc100780 0x7fa5bc101960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c1360700 1 -- 172.21.15.106:0/2360945563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc101140 msgr2=0x7fa5bc101ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c1360700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc101140 0x7fa5bc101ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c1360700 1 -- 172.21.15.106:0/2360945563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc1a6f60 con 0x7fa5bc107200 2024-03-28T02:09:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c0b5f700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5bc100780 0x7fa5bc101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.033+0000 7fa5c1360700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc107200 0x7fa5bc1023e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400ef90 tx=0x7fa5b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.034+0000 7fa5b9ffb700 1 -- 172.21.15.106:0/2360945563 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40092e0 con 0x7fa5bc107200 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.034+0000 7fa5b9ffb700 1 -- 172.21.15.106:0/2360945563 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5b4004bb0 con 0x7fa5bc107200 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.034+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc1a71f0 con 0x7fa5bc107200 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.034+0000 7fa5b9ffb700 1 -- 172.21.15.106:0/2360945563 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4004280 con 0x7fa5bc107200 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.034+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc1a7700 con 0x7fa5bc107200 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.035+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc103aa0 con 0x7fa5bc107200 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.036+0000 7fa5b9ffb700 1 -- 172.21.15.106:0/2360945563 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa5b401b030 con 0x7fa5bc107200 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.036+0000 7fa5b9ffb700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5ac0a2920 0x7fa5ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.036+0000 7fa5c0b5f700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5ac0a2920 0x7fa5ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.036+0000 7fa5b9ffb700 1 -- 172.21.15.106:0/2360945563 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fa5b40cea60 con 0x7fa5bc107200 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.037+0000 7fa5c0b5f700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5ac0a2920 0x7fa5ac0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0007f50 tx=0x7fa5b00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:09:54.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.041+0000 7fa5b9ffb700 1 -- 172.21.15.106:0/2360945563 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa5b4022020 con 0x7fa5bc107200 2024-03-28T02:09:54.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.244+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa5bc02da30 con 0x7fa5bc107200 2024-03-28T02:09:54.238 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.244+0000 7fa5b9ffb700 1 -- 172.21.15.106:0/2360945563 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa5b4010040 con 0x7fa5bc107200 2024-03-28T02:09:54.238 INFO:teuthology.orchestra.run.smithi106.stdout:107374182467 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5ac0a2920 msgr2=0x7fa5ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5ac0a2920 0x7fa5ac0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0007f50 tx=0x7fa5b00073d0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc107200 msgr2=0x7fa5bc1023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc107200 0x7fa5bc1023e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400ef90 tx=0x7fa5b400c3a0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 shutdown_connections 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa5ac0a2920 0x7fa5ac0a4de0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5bc107200 0x7fa5bc1023e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5bc100780 0x7fa5bc101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 --2- 172.21.15.106:0/2360945563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc101140 0x7fa5bc101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.246+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 >> 172.21.15.106:0/2360945563 conn(0x7fa5bc077690 msgr2=0x7fa5bc105140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:09:54.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.247+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 shutdown_connections 2024-03-28T02:09:54.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:09:54.247+0000 7fa5c2dc3700 1 -- 172.21.15.106:0/2360945563 wait complete. 2024-03-28T02:09:54.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:54 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2360945563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T02:09:54.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:54 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2360945563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T02:09:54.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:54 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2360945563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T02:09:55.087 INFO:teuthology.orchestra.run.smithi106.stdout:77309411401 2024-03-28T02:09:55.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T02:09:55.546 INFO:teuthology.orchestra.run.smithi106.stdout:343597383706 2024-03-28T02:09:55.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T02:09:55.591 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:55.592 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:55.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:55 smithi153 ceph-mon[36228]: pgmap v327: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s; 27 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T02:09:55.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:55 smithi106 ceph-mon[31965]: pgmap v327: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s; 27 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T02:09:55.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:55 smithi178 ceph-mon[36521]: pgmap v327: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s; 27 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T02:09:56.093 INFO:teuthology.orchestra.run.smithi106.stdout:219043332143 2024-03-28T02:09:56.093 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T02:09:56.144 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:56.603 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:56.652 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182462 got 107374182467 for osd.3 2024-03-28T02:09:56.652 DEBUG:teuthology.parallel:result is None 2024-03-28T02:09:57.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:57 smithi106 ceph-mon[31965]: pgmap v328: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s; 24 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T02:09:57.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:57 smithi153 ceph-mon[36228]: pgmap v328: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s; 24 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T02:09:57.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:57 smithi178 ceph-mon[36521]: pgmap v328: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s; 24 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T02:09:59.426 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:59.426 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:59.427 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:59.427 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:09:59.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:09:59 smithi153 ceph-mon[36228]: pgmap v329: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T02:09:59.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:09:59 smithi106 ceph-mon[31965]: pgmap v329: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T02:09:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:09:59 smithi178 ceph-mon[36521]: pgmap v329: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T02:10:00.315 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:00.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:00 smithi106 ceph-mon[31965]: overall HEALTH_OK 2024-03-28T02:10:00.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:00 smithi153 ceph-mon[36228]: overall HEALTH_OK 2024-03-28T02:10:00.783 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:00.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:00 smithi178 ceph-mon[36521]: overall HEALTH_OK 2024-03-28T02:10:01.568 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:01 smithi106 ceph-mon[31965]: pgmap v330: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:01 smithi178 ceph-mon[36521]: pgmap v330: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:01 smithi153 ceph-mon[36228]: pgmap v330: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:02.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 -- 172.21.15.106:0/3776915096 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac101090 msgr2=0x7f26ac1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:02.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/3776915096 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac101090 0x7f26ac1014f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f269c009990 tx=0x7f269c009ca0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 -- 172.21.15.106:0/3776915096 shutdown_connections 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/3776915096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ac101a30 0x7f26ac10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/3776915096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ac1070d0 0x7f26ac1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/3776915096 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac101090 0x7f26ac1014f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 -- 172.21.15.106:0/3776915096 >> 172.21.15.106:0/3776915096 conn(0x7f26ac077fc0 msgr2=0x7f26ac0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 -- 172.21.15.106:0/3776915096 shutdown_connections 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 -- 172.21.15.106:0/3776915096 wait complete. 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 Processor -- start 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 -- start start 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.260+0000 7f26b3cdf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ac101090 0x7f26ac199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b3cdf700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ac101a30 0x7f26ac199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b3cdf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac1070d0 0x7f26ac1a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:02.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b3cdf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac111330 con 0x7f26ac1070d0 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b3cdf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac1111b0 con 0x7f26ac101090 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b3cdf700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac1114b0 con 0x7f26ac101a30 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b227c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac1070d0 0x7f26ac1a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b1a7b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ac101090 0x7f26ac199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b227c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac1070d0 0x7f26ac1a47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55052/0 (socket says 172.21.15.106:55052) 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b227c700 1 -- 172.21.15.106:0/2086328211 learned_addr learned my addr 172.21.15.106:0/2086328211 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b127a700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ac101a30 0x7f26ac199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b227c700 1 -- 172.21.15.106:0/2086328211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ac101a30 msgr2=0x7f26ac199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b227c700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ac101a30 0x7f26ac199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b227c700 1 -- 172.21.15.106:0/2086328211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ac101090 msgr2=0x7f26ac199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b227c700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ac101090 0x7f26ac199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b227c700 1 -- 172.21.15.106:0/2086328211 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac1a4d10 con 0x7f26ac1070d0 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b227c700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac1070d0 0x7f26ac1a47d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f26a400ef30 tx=0x7f26a400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:02.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26a2ffd700 1 -- 172.21.15.106:0/2086328211 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a40092e0 con 0x7f26ac1070d0 2024-03-28T02:10:02.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ac1a4ee0 con 0x7f26ac1070d0 2024-03-28T02:10:02.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26a2ffd700 1 -- 172.21.15.106:0/2086328211 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26a4004bb0 con 0x7f26ac1070d0 2024-03-28T02:10:02.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26a2ffd700 1 -- 172.21.15.106:0/2086328211 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4004280 con 0x7f26ac1070d0 2024-03-28T02:10:02.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.261+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ac1a5340 con 0x7f26ac1070d0 2024-03-28T02:10:02.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.262+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ac068e50 con 0x7f26ac1070d0 2024-03-28T02:10:02.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.264+0000 7f26a2ffd700 1 -- 172.21.15.106:0/2086328211 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f26a401b030 con 0x7f26ac1070d0 2024-03-28T02:10:02.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.265+0000 7f26a2ffd700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26980a2920 0x7f26980a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:02.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.265+0000 7f26b1a7b700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26980a2920 0x7f26980a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:02.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.265+0000 7f26a2ffd700 1 -- 172.21.15.106:0/2086328211 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f26a40cf2f0 con 0x7f26ac1070d0 2024-03-28T02:10:02.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.265+0000 7f26b1a7b700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26980a2920 0x7f26980a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f26a8009be0 tx=0x7f26a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:02.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.268+0000 7f26a2ffd700 1 -- 172.21.15.106:0/2086328211 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26a4022020 con 0x7f26ac1070d0 2024-03-28T02:10:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.472+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f26ac02db50 con 0x7f26ac1070d0 2024-03-28T02:10:02.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.472+0000 7f26a2ffd700 1 -- 172.21.15.106:0/2086328211 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f26a4010040 con 0x7f26ac1070d0 2024-03-28T02:10:02.466 INFO:teuthology.orchestra.run.smithi106.stdout:141733920832 2024-03-28T02:10:02.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26980a2920 msgr2=0x7f26980a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26980a2920 0x7f26980a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f26a8009be0 tx=0x7f26a8009260 comp rx=0 tx=0).stop 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac1070d0 msgr2=0x7f26ac1a47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac1070d0 0x7f26ac1a47d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f26a400ef30 tx=0x7f26a400c3a0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 shutdown_connections 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f26980a2920 0x7f26980a4de0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ac1070d0 0x7f26ac1a47d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ac101090 0x7f26ac199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 --2- 172.21.15.106:0/2086328211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ac101a30 0x7f26ac199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 >> 172.21.15.106:0/2086328211 conn(0x7f26ac077fc0 msgr2=0x7f26ac0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 shutdown_connections 2024-03-28T02:10:02.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:02.474+0000 7f26b3cdf700 1 -- 172.21.15.106:0/2086328211 wait complete. 2024-03-28T02:10:02.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2086328211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T02:10:02.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2086328211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T02:10:02.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2086328211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T02:10:03.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:03 smithi106 ceph-mon[31965]: pgmap v331: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:03 smithi178 ceph-mon[36521]: pgmap v331: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:03.924 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920827 got 141733920832 for osd.4 2024-03-28T02:10:03.925 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:03 smithi153 ceph-mon[36228]: pgmap v331: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:05.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:05 smithi106 ceph-mon[31965]: pgmap v332: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-28T02:10:05.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:05 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/507735763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T02:10:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:05 smithi178 ceph-mon[36521]: pgmap v332: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-28T02:10:05.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:05 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/507735763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T02:10:05.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.456+0000 7fe301305700 1 -- 172.21.15.106:0/3204307923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2fc101a30 msgr2=0x7fe2fc10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:05.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.456+0000 7fe301305700 1 --2- 172.21.15.106:0/3204307923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2fc101a30 0x7fe2fc10df10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400b0a0 tx=0x7fe2f400b3b0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.456+0000 7fe301305700 1 -- 172.21.15.106:0/3204307923 shutdown_connections 2024-03-28T02:10:05.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.456+0000 7fe301305700 1 --2- 172.21.15.106:0/3204307923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2fc101a30 0x7fe2fc10df10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.456+0000 7fe301305700 1 --2- 172.21.15.106:0/3204307923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2fc1070d0 0x7fe2fc1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.456+0000 7fe301305700 1 --2- 172.21.15.106:0/3204307923 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2fc101090 0x7fe2fc1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.456+0000 7fe301305700 1 -- 172.21.15.106:0/3204307923 >> 172.21.15.106:0/3204307923 conn(0x7fe2fc077fc0 msgr2=0x7fe2fc0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.456+0000 7fe301305700 1 -- 172.21.15.106:0/3204307923 shutdown_connections 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.456+0000 7fe301305700 1 -- 172.21.15.106:0/3204307923 wait complete. 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe301305700 1 Processor -- start 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe301305700 1 -- start start 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe301305700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2fc101090 0x7fe2fc19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe301305700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2fc101a30 0x7fe2fc19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe301305700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2fc1070d0 0x7fe2fc198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe301305700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc111350 con 0x7fe2fc101a30 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe301305700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc1111d0 con 0x7fe2fc101090 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe301305700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc1114d0 con 0x7fe2fc1070d0 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2fa7fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2fc101a30 0x7fe2fc19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:05.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2fa7fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2fc101a30 0x7fe2fc19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55066/0 (socket says 172.21.15.106:55066) 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2faffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2fc101090 0x7fe2fc19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2fa7fc700 1 -- 172.21.15.106:0/507735763 learned_addr learned my addr 172.21.15.106:0/507735763 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2fb7fe700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2fc1070d0 0x7fe2fc198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2faffd700 1 -- 172.21.15.106:0/507735763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2fc1070d0 msgr2=0x7fe2fc198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2faffd700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2fc1070d0 0x7fe2fc198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2faffd700 1 -- 172.21.15.106:0/507735763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2fc101a30 msgr2=0x7fe2fc19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2faffd700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2fc101a30 0x7fe2fc19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.457+0000 7fe2faffd700 1 -- 172.21.15.106:0/507735763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc199040 con 0x7fe2fc101090 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.458+0000 7fe2faffd700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2fc101090 0x7fe2fc19e690 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000e9f0 tx=0x7fe2f000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.458+0000 7fe2e3fff700 1 -- 172.21.15.106:0/507735763 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f000c9c0 con 0x7fe2fc101090 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.458+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc199330 con 0x7fe2fc101090 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.458+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc1a5610 con 0x7fe2fc101090 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.458+0000 7fe2e3fff700 1 -- 172.21.15.106:0/507735763 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2f000cb60 con 0x7fe2fc101090 2024-03-28T02:10:05.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.458+0000 7fe2e3fff700 1 -- 172.21.15.106:0/507735763 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0018680 con 0x7fe2fc101090 2024-03-28T02:10:05.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.459+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e8005330 con 0x7fe2fc101090 2024-03-28T02:10:05.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.461+0000 7fe2e3fff700 1 -- 172.21.15.106:0/507735763 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe2f000cd10 con 0x7fe2fc101090 2024-03-28T02:10:05.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.461+0000 7fe2e3fff700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2e40a2b20 0x7fe2e40a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:05.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.461+0000 7fe2e3fff700 1 -- 172.21.15.106:0/507735763 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fe2f0018420 con 0x7fe2fc101090 2024-03-28T02:10:05.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.461+0000 7fe2fa7fc700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2e40a2b20 0x7fe2e40a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:05.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.462+0000 7fe2fa7fc700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2e40a2b20 0x7fe2e40a4fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc199f20 tx=0x7fe2ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:05.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.469+0000 7fe2e3fff700 1 -- 172.21.15.106:0/507735763 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe2f009b810 con 0x7fe2fc101090 2024-03-28T02:10:05.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.681+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fe2e80058f0 con 0x7fe2fc101090 2024-03-28T02:10:05.902 INFO:teuthology.orchestra.run.smithi106.stdout:309237645346 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.682+0000 7fe2e3fff700 1 -- 172.21.15.106:0/507735763 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fe2f009b110 con 0x7fe2fc101090 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2e40a2b20 msgr2=0x7fe2e40a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2e40a2b20 0x7fe2e40a4fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc199f20 tx=0x7fe2ec009260 comp rx=0 tx=0).stop 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2fc101090 msgr2=0x7fe2fc19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2fc101090 0x7fe2fc19e690 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000e9f0 tx=0x7fe2f000ed00 comp rx=0 tx=0).stop 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 shutdown_connections 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2fc1070d0 0x7fe2fc198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2fc101090 0x7fe2fc19e690 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2e40a2b20 0x7fe2e40a4fe0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 --2- 172.21.15.106:0/507735763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2fc101a30 0x7fe2fc19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 >> 172.21.15.106:0/507735763 conn(0x7fe2fc077fc0 msgr2=0x7fe2fc100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 shutdown_connections 2024-03-28T02:10:05.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:04.684+0000 7fe301305700 1 -- 172.21.15.106:0/507735763 wait complete. 2024-03-28T02:10:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:05 smithi153 ceph-mon[36228]: pgmap v332: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-28T02:10:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:05 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/507735763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T02:10:06.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.562+0000 7ff588abd700 1 -- 172.21.15.106:0/4208856408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 msgr2=0x7ff5840747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:06.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.562+0000 7ff588abd700 1 --2- 172.21.15.106:0/4208856408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 0x7ff5840747a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff574009990 tx=0x7ff574009ca0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.562+0000 7ff588abd700 1 -- 172.21.15.106:0/4208856408 shutdown_connections 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.562+0000 7ff588abd700 1 --2- 172.21.15.106:0/4208856408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff58410ebf0 0x7ff584110fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.562+0000 7ff588abd700 1 --2- 172.21.15.106:0/4208856408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 0x7ff5840747a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.562+0000 7ff588abd700 1 --2- 172.21.15.106:0/4208856408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff584074ce0 0x7ff58410e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.562+0000 7ff588abd700 1 -- 172.21.15.106:0/4208856408 >> 172.21.15.106:0/4208856408 conn(0x7ff584079260 msgr2=0x7ff584079670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.562+0000 7ff588abd700 1 -- 172.21.15.106:0/4208856408 shutdown_connections 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.562+0000 7ff588abd700 1 -- 172.21.15.106:0/4208856408 wait complete. 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff588abd700 1 Processor -- start 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff588abd700 1 -- start start 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff588abd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 0x7ff5841a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff588abd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff584074ce0 0x7ff5841a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff588abd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff58410ebf0 0x7ff58419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff588abd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5841146a0 con 0x7ff5840743c0 2024-03-28T02:10:06.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff588abd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff584114520 con 0x7ff584074ce0 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff588abd700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff584114820 con 0x7ff58410ebf0 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff58259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 0x7ff5841a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff58259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 0x7ff5841a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55112/0 (socket says 172.21.15.106:55112) 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff58259c700 1 -- 172.21.15.106:0/417959695 learned_addr learned my addr 172.21.15.106:0/417959695 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff582d9d700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff58410ebf0 0x7ff58419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.563+0000 7ff579d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff584074ce0 0x7ff5841a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff58259c700 1 -- 172.21.15.106:0/417959695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff58410ebf0 msgr2=0x7ff58419a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff58259c700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff58410ebf0 0x7ff58419a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff58259c700 1 -- 172.21.15.106:0/417959695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff584074ce0 msgr2=0x7ff5841a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff58259c700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff584074ce0 0x7ff5841a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff58259c700 1 -- 172.21.15.106:0/417959695 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58419b080 con 0x7ff5840743c0 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff582d9d700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff58410ebf0 0x7ff58419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:06.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff58259c700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 0x7ff5841a0760 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff574000c00 tx=0x7ff57400bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:06.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff57bfff700 1 -- 172.21.15.106:0/417959695 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574027070 con 0x7ff5840743c0 2024-03-28T02:10:06.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58419b310 con 0x7ff5840743c0 2024-03-28T02:10:06.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff57bfff700 1 -- 172.21.15.106:0/417959695 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff574003d70 con 0x7ff5840743c0 2024-03-28T02:10:06.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff57bfff700 1 -- 172.21.15.106:0/417959695 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57400f460 con 0x7ff5840743c0 2024-03-28T02:10:06.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.564+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5841a7630 con 0x7ff5840743c0 2024-03-28T02:10:06.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.565+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff584068e50 con 0x7ff5840743c0 2024-03-28T02:10:06.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.567+0000 7ff57bfff700 1 -- 172.21.15.106:0/417959695 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ff57402a070 con 0x7ff5840743c0 2024-03-28T02:10:06.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.567+0000 7ff57bfff700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2920 0x7ff5700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:06.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.568+0000 7ff579d9b700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2920 0x7ff5700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:06.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.568+0000 7ff57bfff700 1 -- 172.21.15.106:0/417959695 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7ff5740d0050 con 0x7ff5840743c0 2024-03-28T02:10:06.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.568+0000 7ff579d9b700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2920 0x7ff5700a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff5841107d0 tx=0x7ff56c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:06.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.571+0000 7ff57bfff700 1 -- 172.21.15.106:0/417959695 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff574013070 con 0x7ff5840743c0 2024-03-28T02:10:06.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:06 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4157493340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T02:10:06.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.774+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7ff58402da30 con 0x7ff5840743c0 2024-03-28T02:10:06.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.774+0000 7ff57bfff700 1 -- 172.21.15.106:0/417959695 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff57409cce0 con 0x7ff5840743c0 2024-03-28T02:10:06.768 INFO:teuthology.orchestra.run.smithi106.stdout:253403070509 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.776+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2920 msgr2=0x7ff5700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.776+0000 7ff588abd700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2920 0x7ff5700a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff5841107d0 tx=0x7ff56c009260 comp rx=0 tx=0).stop 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.776+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 msgr2=0x7ff5841a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.776+0000 7ff588abd700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 0x7ff5841a0760 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff574000c00 tx=0x7ff57400bab0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.777+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 shutdown_connections 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.777+0000 7ff588abd700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff58410ebf0 0x7ff58419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.777+0000 7ff588abd700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2920 0x7ff5700a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.777+0000 7ff588abd700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5840743c0 0x7ff5841a0760 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.777+0000 7ff588abd700 1 --2- 172.21.15.106:0/417959695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff584074ce0 0x7ff5841a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:06.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.777+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 >> 172.21.15.106:0/417959695 conn(0x7ff584079260 msgr2=0x7ff5840ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:06.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.777+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 shutdown_connections 2024-03-28T02:10:06.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:06.777+0000 7ff588abd700 1 -- 172.21.15.106:0/417959695 wait complete. 2024-03-28T02:10:06.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:06 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4157493340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T02:10:06.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:06 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4157493340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T02:10:07.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:07 smithi106 ceph-mon[31965]: pgmap v333: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:07.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:07 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/417959695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T02:10:07.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:07 smithi178 ceph-mon[36521]: pgmap v333: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:07.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:07 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/417959695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T02:10:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:07 smithi153 ceph-mon[36228]: pgmap v333: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:07 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/417959695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T02:10:08.440 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:08.976 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645339 got 309237645346 for osd.10 2024-03-28T02:10:08.976 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:09.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:09 smithi106 ceph-mon[31965]: pgmap v334: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:10:09.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:09 smithi178 ceph-mon[36521]: pgmap v334: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:10:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:09 smithi153 ceph-mon[36228]: pgmap v334: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:10:10.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.655+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4202254207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006b440 msgr2=0x7f7f90106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:10.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.655+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4202254207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006b440 0x7f7f90106dd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7f800099c0 tx=0x7f7f80009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.655+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4202254207 shutdown_connections 2024-03-28T02:10:10.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.655+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4202254207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f90107500 0x7f7f9010f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.655+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4202254207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f9006aa90 0x7f7f9006ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.655+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4202254207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006b440 0x7f7f90106dd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.655+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4202254207 >> 172.21.15.106:0/4202254207 conn(0x7f7f90078000 msgr2=0x7f7f90078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:10.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.655+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4202254207 shutdown_connections 2024-03-28T02:10:10.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.655+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4202254207 wait complete. 2024-03-28T02:10:10.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f97cbf700 1 Processor -- start 2024-03-28T02:10:10.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f97cbf700 1 -- start start 2024-03-28T02:10:10.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f97cbf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006aa90 0x7f7f90101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:10.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f97cbf700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f9006b440 0x7f7f90101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:10.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f97cbf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f90107500 0x7f7f90101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:10.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f97cbf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f90112470 con 0x7f7f9006aa90 2024-03-28T02:10:10.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f97cbf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f901122f0 con 0x7f7f90107500 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f97cbf700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f901125f0 con 0x7f7f9006b440 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f95a5b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006aa90 0x7f7f90101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f95a5b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006aa90 0x7f7f90101150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55090/0 (socket says 172.21.15.106:55090) 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f95a5b700 1 -- 172.21.15.106:0/4157493340 learned_addr learned my addr 172.21.15.106:0/4157493340 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f9525a700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f9006b440 0x7f7f90101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f9625c700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f90107500 0x7f7f90101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f95a5b700 1 -- 172.21.15.106:0/4157493340 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f9006b440 msgr2=0x7f7f90101690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f95a5b700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f9006b440 0x7f7f90101690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f95a5b700 1 -- 172.21.15.106:0/4157493340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f90107500 msgr2=0x7f7f90101bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f95a5b700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f90107500 0x7f7f90101bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.689 INFO:teuthology.orchestra.run.smithi106.stdout:34359738453 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f95a5b700 1 -- 172.21.15.106:0/4157493340 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f901a6eb0 con 0x7f7f9006aa90 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f9525a700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f9006b440 0x7f7f90101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.656+0000 7f7f95a5b700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006aa90 0x7f7f90101150 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e9c0 tx=0x7f7f8c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.657+0000 7f7f86ffd700 1 -- 172.21.15.106:0/4157493340 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00cbd0 con 0x7f7f9006aa90 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.657+0000 7f7f86ffd700 1 -- 172.21.15.106:0/4157493340 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c00ee60 con 0x7f7f9006aa90 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.657+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f901a71a0 con 0x7f7f9006aa90 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.657+0000 7f7f86ffd700 1 -- 172.21.15.106:0/4157493340 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c005630 con 0x7f7f9006aa90 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.657+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f901a7660 con 0x7f7f9006aa90 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.658+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9010a6b0 con 0x7f7f9006aa90 2024-03-28T02:10:10.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.659+0000 7f7f86ffd700 1 -- 172.21.15.106:0/4157493340 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7f8c004750 con 0x7f7f9006aa90 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.659+0000 7f7f86ffd700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f7c0a2920 0x7f7f7c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.659+0000 7f7f9525a700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f7c0a2920 0x7f7f7c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.659+0000 7f7f86ffd700 1 -- 172.21.15.106:0/4157493340 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f7f8c0cf3e0 con 0x7f7f9006aa90 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.660+0000 7f7f9525a700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f7c0a2920 0x7f7f7c0a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009990 tx=0x7f7f80005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.663+0000 7f7f86ffd700 1 -- 172.21.15.106:0/4157493340 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f8c053740 con 0x7f7f9006aa90 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.870+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7f9010a090 con 0x7f7f9006aa90 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.870+0000 7f7f86ffd700 1 -- 172.21.15.106:0/4157493340 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7f8c09bfe0 con 0x7f7f9006aa90 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f7c0a2920 msgr2=0x7f7f7c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f7c0a2920 0x7f7f7c0a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009990 tx=0x7f7f80005fb0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006aa90 msgr2=0x7f7f90101150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006aa90 0x7f7f90101150 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e9c0 tx=0x7f7f8c00ecd0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 shutdown_connections 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f90107500 0x7f7f90101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f7c0a2920 0x7f7f7c0a4de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9006aa90 0x7f7f90101150 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 --2- 172.21.15.106:0/4157493340 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f9006b440 0x7f7f90101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:10.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 >> 172.21.15.106:0/4157493340 conn(0x7f7f90078000 msgr2=0x7f7f900ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:10.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 shutdown_connections 2024-03-28T02:10:10.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:05.872+0000 7f7f97cbf700 1 -- 172.21.15.106:0/4157493340 wait complete. 2024-03-28T02:10:11.581 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738453 for osd.0 2024-03-28T02:10:11.581 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:11.637 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:11 smithi106 ceph-mon[31965]: pgmap v335: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:11.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.747+0000 7f7f55bbc700 1 -- 172.21.15.106:0/2641192666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 msgr2=0x7f7f50101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:11.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.747+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/2641192666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 0x7f7f50101700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009990 tx=0x7f7f40009ca0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.747+0000 7f7f55bbc700 1 -- 172.21.15.106:0/2641192666 shutdown_connections 2024-03-28T02:10:11.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.747+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/2641192666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f501063b0 0x7f7f5010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.748+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/2641192666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 0x7f7f50101700 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.748+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/2641192666 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f50101cd0 0x7f7f50105cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.748+0000 7f7f55bbc700 1 -- 172.21.15.106:0/2641192666 >> 172.21.15.106:0/2641192666 conn(0x7f7f500770b0 msgr2=0x7f7f500774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:11.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.748+0000 7f7f55bbc700 1 -- 172.21.15.106:0/2641192666 shutdown_connections 2024-03-28T02:10:11.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.748+0000 7f7f55bbc700 1 -- 172.21.15.106:0/2641192666 wait complete. 2024-03-28T02:10:11.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.748+0000 7f7f55bbc700 1 Processor -- start 2024-03-28T02:10:11.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.748+0000 7f7f55bbc700 1 -- start start 2024-03-28T02:10:11.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.748+0000 7f7f55bbc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 0x7f7f5019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:11.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.748+0000 7f7f55bbc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f50101cd0 0x7f7f5019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:11.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f55bbc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f501063b0 0x7f7f50198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:11.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f55bbc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f50111370 con 0x7f7f50101320 2024-03-28T02:10:11.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f55bbc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f501111f0 con 0x7f7f50101cd0 2024-03-28T02:10:11.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f55bbc700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f501114f0 con 0x7f7f501063b0 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4f7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 0x7f7f5019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4f7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 0x7f7f5019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51900/0 (socket says 172.21.15.106:51900) 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4f7fe700 1 -- 172.21.15.106:0/1159212654 learned_addr learned my addr 172.21.15.106:0/1159212654 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4effd700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f50101cd0 0x7f7f5019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4ffff700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f501063b0 0x7f7f50198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4f7fe700 1 -- 172.21.15.106:0/1159212654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f501063b0 msgr2=0x7f7f50198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4f7fe700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f501063b0 0x7f7f50198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4f7fe700 1 -- 172.21.15.106:0/1159212654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f50101cd0 msgr2=0x7f7f5019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4f7fe700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f50101cd0 0x7f7f5019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4f7fe700 1 -- 172.21.15.106:0/1159212654 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f50199030 con 0x7f7f50101320 2024-03-28T02:10:11.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4ffff700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f501063b0 0x7f7f50198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:11.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4f7fe700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 0x7f7f5019e680 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7f40000c00 tx=0x7f7f4000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:11.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4cff9700 1 -- 172.21.15.106:0/1159212654 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40027070 con 0x7f7f50101320 2024-03-28T02:10:11.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f501992c0 con 0x7f7f50101320 2024-03-28T02:10:11.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4cff9700 1 -- 172.21.15.106:0/1159212654 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f40003d70 con 0x7f7f50101320 2024-03-28T02:10:11.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f4cff9700 1 -- 172.21.15.106:0/1159212654 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000f460 con 0x7f7f50101320 2024-03-28T02:10:11.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.749+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f501a5600 con 0x7f7f50101320 2024-03-28T02:10:11.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.751+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f50199620 con 0x7f7f50101320 2024-03-28T02:10:11.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.751+0000 7f7f4cff9700 1 -- 172.21.15.106:0/1159212654 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7f4002a070 con 0x7f7f50101320 2024-03-28T02:10:11.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.752+0000 7f7f4cff9700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f380a2920 0x7f7f380a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:11.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.752+0000 7f7f4effd700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f380a2920 0x7f7f380a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:11.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.752+0000 7f7f4cff9700 1 -- 172.21.15.106:0/1159212654 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f7f400cfcd0 con 0x7f7f50101320 2024-03-28T02:10:11.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.752+0000 7f7f4effd700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f380a2920 0x7f7f380a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7f50199f10 tx=0x7f7f44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:11.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.756+0000 7f7f4cff9700 1 -- 172.21.15.106:0/1159212654 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f40013070 con 0x7f7f50101320 2024-03-28T02:10:11.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:11 smithi178 ceph-mon[36521]: pgmap v335: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:11.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.958+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7f50068e50 con 0x7f7f50101320 2024-03-28T02:10:11.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.958+0000 7f7f4cff9700 1 -- 172.21.15.106:0/1159212654 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7f4009c960 con 0x7f7f50101320 2024-03-28T02:10:11.951 INFO:teuthology.orchestra.run.smithi106.stdout:77309411404 2024-03-28T02:10:11.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f380a2920 msgr2=0x7f7f380a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:11.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f380a2920 0x7f7f380a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7f50199f10 tx=0x7f7f44009260 comp rx=0 tx=0).stop 2024-03-28T02:10:11.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 msgr2=0x7f7f5019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:11.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 0x7f7f5019e680 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7f40000c00 tx=0x7f7f4000bab0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 shutdown_connections 2024-03-28T02:10:11.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f501063b0 0x7f7f50198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7f380a2920 0x7f7f380a4de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f50101320 0x7f7f5019e680 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 --2- 172.21.15.106:0/1159212654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f50101cd0 0x7f7f5019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:11.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.960+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 >> 172.21.15.106:0/1159212654 conn(0x7f7f500770b0 msgr2=0x7f7f5006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:11.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.961+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 shutdown_connections 2024-03-28T02:10:11.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:11.961+0000 7f7f55bbc700 1 -- 172.21.15.106:0/1159212654 wait complete. 2024-03-28T02:10:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:11 smithi153 ceph-mon[36228]: pgmap v335: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:12.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:12 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1159212654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T02:10:12.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:12 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1159212654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T02:10:12.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:12 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1159212654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T02:10:13.294 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 -- 172.21.15.106:0/2738832604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e4101cd0 msgr2=0x7fb9e4105cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:13.294 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2738832604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e4101cd0 0x7fb9e4105cd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009990 tx=0x7fb9d4009ca0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.294 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 -- 172.21.15.106:0/2738832604 shutdown_connections 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2738832604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9e41063b0 0x7fb9e410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2738832604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4101320 0x7fb9e4101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2738832604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e4101cd0 0x7fb9e4105cd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 -- 172.21.15.106:0/2738832604 >> 172.21.15.106:0/2738832604 conn(0x7fb9e40770b0 msgr2=0x7fb9e40774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 -- 172.21.15.106:0/2738832604 shutdown_connections 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 -- 172.21.15.106:0/2738832604 wait complete. 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 Processor -- start 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.277+0000 7fb9ec587700 1 -- start start 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9ec587700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9e4101320 0x7fb9e41997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9ec587700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4101cd0 0x7fb9e4199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9ec587700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e41063b0 0x7fb9e41a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9ec587700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e4111370 con 0x7fb9e41063b0 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9ec587700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e41111f0 con 0x7fb9e4101cd0 2024-03-28T02:10:13.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9ec587700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e41114f0 con 0x7fb9e4101320 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9eab24700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e41063b0 0x7fb9e41a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9eab24700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e41063b0 0x7fb9e41a4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51914/0 (socket says 172.21.15.106:51914) 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9eab24700 1 -- 172.21.15.106:0/2397489365 learned_addr learned my addr 172.21.15.106:0/2397489365 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9ea323700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9e4101320 0x7fb9e41997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9e9b22700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4101cd0 0x7fb9e4199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9eab24700 1 -- 172.21.15.106:0/2397489365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9e4101320 msgr2=0x7fb9e41997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9eab24700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9e4101320 0x7fb9e41997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9eab24700 1 -- 172.21.15.106:0/2397489365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4101cd0 msgr2=0x7fb9e4199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9eab24700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4101cd0 0x7fb9e4199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9eab24700 1 -- 172.21.15.106:0/2397489365 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e41a4d70 con 0x7fb9e41063b0 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9eab24700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e41063b0 0x7fb9e41a4830 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00ef30 tx=0x7fb9dc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9db7fe700 1 -- 172.21.15.106:0/2397489365 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0092e0 con 0x7fb9e41063b0 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9db7fe700 1 -- 172.21.15.106:0/2397489365 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9dc004bb0 con 0x7fb9e41063b0 2024-03-28T02:10:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e41a4f40 con 0x7fb9e41063b0 2024-03-28T02:10:13.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9db7fe700 1 -- 172.21.15.106:0/2397489365 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc004280 con 0x7fb9e41063b0 2024-03-28T02:10:13.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.278+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e41a5340 con 0x7fb9e41063b0 2024-03-28T02:10:13.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.279+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e41027b0 con 0x7fb9e41063b0 2024-03-28T02:10:13.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.281+0000 7fb9db7fe700 1 -- 172.21.15.106:0/2397489365 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb9dc01b030 con 0x7fb9e41063b0 2024-03-28T02:10:13.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.282+0000 7fb9db7fe700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb9d00a2920 0x7fb9d00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:13.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.282+0000 7fb9ea323700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb9d00a2920 0x7fb9d00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:13.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.282+0000 7fb9db7fe700 1 -- 172.21.15.106:0/2397489365 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fb9dc0cf280 con 0x7fb9e41063b0 2024-03-28T02:10:13.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.282+0000 7fb9ea323700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb9d00a2920 0x7fb9d00a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0009c60 tx=0x7fb9e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:13.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.285+0000 7fb9db7fe700 1 -- 172.21.15.106:0/2397489365 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9dc022020 con 0x7fb9e41063b0 2024-03-28T02:10:13.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.491+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb9e402dbf0 con 0x7fb9e41063b0 2024-03-28T02:10:13.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.491+0000 7fb9db7fe700 1 -- 172.21.15.106:0/2397489365 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb9dc010040 con 0x7fb9e41063b0 2024-03-28T02:10:13.485 INFO:teuthology.orchestra.run.smithi106.stdout:55834574929 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.493+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb9d00a2920 msgr2=0x7fb9d00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.493+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb9d00a2920 0x7fb9d00a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0009c60 tx=0x7fb9e0009260 comp rx=0 tx=0).stop 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.493+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e41063b0 msgr2=0x7fb9e41a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.493+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e41063b0 0x7fb9e41a4830 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00ef30 tx=0x7fb9dc00c3a0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.494+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 shutdown_connections 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.494+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb9d00a2920 0x7fb9d00a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.494+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e41063b0 0x7fb9e41a4830 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.494+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9e4101320 0x7fb9e41997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.494+0000 7fb9ec587700 1 --2- 172.21.15.106:0/2397489365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4101cd0 0x7fb9e4199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.494+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 >> 172.21.15.106:0/2397489365 conn(0x7fb9e40770b0 msgr2=0x7fb9e4100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.494+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 shutdown_connections 2024-03-28T02:10:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:13.494+0000 7fb9ec587700 1 -- 172.21.15.106:0/2397489365 wait complete. 2024-03-28T02:10:13.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:13 smithi106 ceph-mon[31965]: pgmap v336: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:13.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:13 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2397489365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T02:10:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:13 smithi178 ceph-mon[36521]: pgmap v336: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:13 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2397489365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T02:10:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:13 smithi153 ceph-mon[36228]: pgmap v336: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:13 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2397489365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T02:10:14.220 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574924 got 55834574929 for osd.1 2024-03-28T02:10:14.220 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:14.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.463+0000 7f1b7afce700 1 -- 172.21.15.106:0/862802453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b741081c0 msgr2=0x7f1b741085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:14.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.463+0000 7f1b7afce700 1 --2- 172.21.15.106:0/862802453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b741081c0 0x7f1b741085a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1b680099c0 tx=0x7f1b68009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.463+0000 7f1b7afce700 1 -- 172.21.15.106:0/862802453 shutdown_connections 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.463+0000 7f1b7afce700 1 --2- 172.21.15.106:0/862802453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b74102ba0 0x7f1b7410efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.463+0000 7f1b7afce700 1 --2- 172.21.15.106:0/862802453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b741081c0 0x7f1b741085a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.463+0000 7f1b7afce700 1 --2- 172.21.15.106:0/862802453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b741021e0 0x7f1b74102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.463+0000 7f1b7afce700 1 -- 172.21.15.106:0/862802453 >> 172.21.15.106:0/862802453 conn(0x7f1b740792c0 msgr2=0x7f1b740796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.463+0000 7f1b7afce700 1 -- 172.21.15.106:0/862802453 shutdown_connections 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.463+0000 7f1b7afce700 1 -- 172.21.15.106:0/862802453 wait complete. 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b7afce700 1 Processor -- start 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b7afce700 1 -- start start 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b7afce700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b741021e0 0x7f1b7419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b7afce700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b74102ba0 0x7f1b7419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b7afce700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b741081c0 0x7f1b74198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b7afce700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b74112440 con 0x7f1b74102ba0 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b7afce700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b741122c0 con 0x7f1b741021e0 2024-03-28T02:10:14.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b7afce700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b741125c0 con 0x7f1b741081c0 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b78d6a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b741021e0 0x7f1b7419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b73fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b74102ba0 0x7f1b7419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b78d6a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b741021e0 0x7f1b7419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:43438/0 (socket says 172.21.15.106:43438) 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b78d6a700 1 -- 172.21.15.106:0/3991123388 learned_addr learned my addr 172.21.15.106:0/3991123388 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.464+0000 7f1b7956b700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b741081c0 0x7f1b74198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b73fff700 1 -- 172.21.15.106:0/3991123388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b741081c0 msgr2=0x7f1b74198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b73fff700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b741081c0 0x7f1b74198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b73fff700 1 -- 172.21.15.106:0/3991123388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b741021e0 msgr2=0x7f1b7419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b73fff700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b741021e0 0x7f1b7419e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b73fff700 1 -- 172.21.15.106:0/3991123388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b74199000 con 0x7f1b74102ba0 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b78d6a700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b741021e0 0x7f1b7419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b73fff700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b74102ba0 0x7f1b7419eb90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000e9f0 tx=0x7f1b6000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b71ffb700 1 -- 172.21.15.106:0/3991123388 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6000c9c0 con 0x7f1b74102ba0 2024-03-28T02:10:14.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b71ffb700 1 -- 172.21.15.106:0/3991123388 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b6000cb60 con 0x7f1b74102ba0 2024-03-28T02:10:14.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b741992f0 con 0x7f1b74102ba0 2024-03-28T02:10:14.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b71ffb700 1 -- 172.21.15.106:0/3991123388 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60018680 con 0x7f1b74102ba0 2024-03-28T02:10:14.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.465+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b741a55d0 con 0x7f1b74102ba0 2024-03-28T02:10:14.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.466+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b74103870 con 0x7f1b74102ba0 2024-03-28T02:10:14.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.468+0000 7f1b71ffb700 1 -- 172.21.15.106:0/3991123388 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1b60010430 con 0x7f1b74102ba0 2024-03-28T02:10:14.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.468+0000 7f1b71ffb700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1b640a2910 0x7f1b640a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:14.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.468+0000 7f1b78d6a700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1b640a2910 0x7f1b640a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:14.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.468+0000 7f1b71ffb700 1 -- 172.21.15.106:0/3991123388 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f1b600cf040 con 0x7f1b74102ba0 2024-03-28T02:10:14.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.469+0000 7f1b78d6a700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1b640a2910 0x7f1b640a4dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1b68003800 tx=0x7f1b68005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:14.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.472+0000 7f1b71ffb700 1 -- 172.21.15.106:0/3991123388 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1b6009bda0 con 0x7f1b74102ba0 2024-03-28T02:10:14.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.674+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1b74199c50 con 0x7f1b74102ba0 2024-03-28T02:10:14.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.675+0000 7f1b71ffb700 1 -- 172.21.15.106:0/3991123388 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1b6009b6a0 con 0x7f1b74102ba0 2024-03-28T02:10:14.668 INFO:teuthology.orchestra.run.smithi106.stdout:163208757310 2024-03-28T02:10:14.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1b640a2910 msgr2=0x7f1b640a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:14.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1b640a2910 0x7f1b640a4dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1b68003800 tx=0x7f1b68005960 comp rx=0 tx=0).stop 2024-03-28T02:10:14.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b74102ba0 msgr2=0x7f1b7419eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:14.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b74102ba0 0x7f1b7419eb90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000e9f0 tx=0x7f1b6000ed00 comp rx=0 tx=0).stop 2024-03-28T02:10:14.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 shutdown_connections 2024-03-28T02:10:14.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b741081c0 0x7f1b74198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b741021e0 0x7f1b7419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f1b640a2910 0x7f1b640a4dd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 --2- 172.21.15.106:0/3991123388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b74102ba0 0x7f1b7419eb90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:14.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 >> 172.21.15.106:0/3991123388 conn(0x7f1b740792c0 msgr2=0x7f1b74100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:14.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 shutdown_connections 2024-03-28T02:10:14.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:14.677+0000 7f1b7afce700 1 -- 172.21.15.106:0/3991123388 wait complete. 2024-03-28T02:10:14.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:14 smithi106 ceph-mon[31965]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T02:10:14.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:14 smithi178 ceph-mon[36521]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T02:10:14.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:14 smithi153 ceph-mon[36228]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T02:10:15.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.510+0000 7f9d38833700 1 -- 172.21.15.106:0/1984577342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 msgr2=0x7f9d3410f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:15.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.510+0000 7f9d38833700 1 --2- 172.21.15.106:0/1984577342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 0x7f9d3410f060 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009fc0 tx=0x7f9d2c00b330 comp rx=0 tx=0).stop 2024-03-28T02:10:15.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.510+0000 7f9d38833700 1 -- 172.21.15.106:0/1984577342 shutdown_connections 2024-03-28T02:10:15.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.510+0000 7f9d38833700 1 --2- 172.21.15.106:0/1984577342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 0x7f9d3410f060 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:15.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.510+0000 7f9d38833700 1 --2- 172.21.15.106:0/1984577342 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d34108260 0x7f9d34108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:15.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.510+0000 7f9d38833700 1 --2- 172.21.15.106:0/1984577342 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d34102210 0x7f9d34102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:15.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.510+0000 7f9d38833700 1 -- 172.21.15.106:0/1984577342 >> 172.21.15.106:0/1984577342 conn(0x7f9d340792e0 msgr2=0x7f9d340796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:15.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.510+0000 7f9d38833700 1 -- 172.21.15.106:0/1984577342 shutdown_connections 2024-03-28T02:10:15.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.510+0000 7f9d38833700 1 -- 172.21.15.106:0/1984577342 wait complete. 2024-03-28T02:10:15.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d38833700 1 Processor -- start 2024-03-28T02:10:15.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d38833700 1 -- start start 2024-03-28T02:10:15.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d38833700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d34102210 0x7f9d3419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:15.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d38833700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 0x7f9d3419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:15.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d38833700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d34108260 0x7f9d34198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:15.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d38833700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d34112480 con 0x7f9d34108260 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d38833700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d34112300 con 0x7f9d34102bd0 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d38833700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d34112600 con 0x7f9d34102210 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d32d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d34108260 0x7f9d34198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d31d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 0x7f9d3419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d31d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 0x7f9d3419eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:43468/0 (socket says 172.21.15.106:43468) 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d3259c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d34102210 0x7f9d3419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d32d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d34108260 0x7f9d34198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51952/0 (socket says 172.21.15.106:51952) 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.511+0000 7f9d31d9b700 1 -- 172.21.15.106:0/3508205292 learned_addr learned my addr 172.21.15.106:0/3508205292 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d31d9b700 1 -- 172.21.15.106:0/3508205292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d34102210 msgr2=0x7f9d3419e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d31d9b700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d34102210 0x7f9d3419e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d31d9b700 1 -- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d34108260 msgr2=0x7f9d34198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d31d9b700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d34108260 0x7f9d34198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d31d9b700 1 -- 172.21.15.106:0/3508205292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d34199000 con 0x7f9d34102bd0 2024-03-28T02:10:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d3259c700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d34102210 0x7f9d3419e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:15.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d32d9d700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d34108260 0x7f9d34198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:15.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d31d9b700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 0x7f9d3419eb40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800ec10 tx=0x7f9d2800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:15.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d237fe700 1 -- 172.21.15.106:0/3508205292 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800cc90 con 0x7f9d34102bd0 2024-03-28T02:10:15.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d237fe700 1 -- 172.21.15.106:0/3508205292 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d2800ce30 con 0x7f9d34102bd0 2024-03-28T02:10:15.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d341992f0 con 0x7f9d34102bd0 2024-03-28T02:10:15.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d237fe700 1 -- 172.21.15.106:0/3508205292 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28010690 con 0x7f9d34102bd0 2024-03-28T02:10:15.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.512+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d341a5580 con 0x7f9d34102bd0 2024-03-28T02:10:15.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.513+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d341038d0 con 0x7f9d34102bd0 2024-03-28T02:10:15.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.514+0000 7f9d237fe700 1 -- 172.21.15.106:0/3508205292 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9d28003680 con 0x7f9d34102bd0 2024-03-28T02:10:15.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.515+0000 7f9d237fe700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9d1c0a2890 0x7f9d1c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:15.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.515+0000 7f9d3259c700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9d1c0a2890 0x7f9d1c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:15.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.515+0000 7f9d237fe700 1 -- 172.21.15.106:0/3508205292 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f9d280cf1f0 con 0x7f9d34102bd0 2024-03-28T02:10:15.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.515+0000 7f9d3259c700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9d1c0a2890 0x7f9d1c0a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009fd0 tx=0x7f9d24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:15.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.518+0000 7f9d237fe700 1 -- 172.21.15.106:0/3508205292 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9d2801f340 con 0x7f9d34102bd0 2024-03-28T02:10:15.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.726+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f9d34199c00 con 0x7f9d34102bd0 2024-03-28T02:10:15.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:15 smithi106 ceph-mon[31965]: pgmap v337: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T02:10:15.720 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:15 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3991123388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T02:10:15.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.726+0000 7f9d237fe700 1 -- 172.21.15.106:0/3508205292 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9d2809bfe0 con 0x7f9d34102bd0 2024-03-28T02:10:15.720 INFO:teuthology.orchestra.run.smithi106.stdout:287762808874 2024-03-28T02:10:15.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9d1c0a2890 msgr2=0x7f9d1c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:15.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9d1c0a2890 0x7f9d1c0a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009fd0 tx=0x7f9d24009260 comp rx=0 tx=0).stop 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 msgr2=0x7f9d3419eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 0x7f9d3419eb40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800ec10 tx=0x7f9d2800ef20 comp rx=0 tx=0).stop 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 shutdown_connections 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f9d1c0a2890 0x7f9d1c0a4d50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d34108260 0x7f9d34198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d34102210 0x7f9d3419e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 --2- 172.21.15.106:0/3508205292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d34102bd0 0x7f9d3419eb40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 >> 172.21.15.106:0/3508205292 conn(0x7f9d340792e0 msgr2=0x7f9d34100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 shutdown_connections 2024-03-28T02:10:15.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.729+0000 7f9d38833700 1 -- 172.21.15.106:0/3508205292 wait complete. 2024-03-28T02:10:15.772 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:15 smithi178 ceph-mon[36521]: pgmap v337: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T02:10:15.772 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:15 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3991123388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T02:10:15.959 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070506 got 253403070509 for osd.8 2024-03-28T02:10:15.959 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:15.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:15 smithi153 ceph-mon[36228]: pgmap v337: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T02:10:15.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:15 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3991123388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T02:10:16.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.412+0000 7f31b5625700 1 -- 172.21.15.106:0/187546493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b01024a0 msgr2=0x7f31b0105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:16.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.412+0000 7f31b5625700 1 --2- 172.21.15.106:0/187546493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b01024a0 0x7f31b0105d30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f31a40099c0 tx=0x7f31a4009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.412+0000 7f31b5625700 1 -- 172.21.15.106:0/187546493 shutdown_connections 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.412+0000 7f31b5625700 1 --2- 172.21.15.106:0/187546493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b0106410 0x7f31b010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.412+0000 7f31b5625700 1 --2- 172.21.15.106:0/187546493 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b0101af0 0x7f31b0101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.412+0000 7f31b5625700 1 --2- 172.21.15.106:0/187546493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b01024a0 0x7f31b0105d30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.412+0000 7f31b5625700 1 -- 172.21.15.106:0/187546493 >> 172.21.15.106:0/187546493 conn(0x7f31b0077690 msgr2=0x7f31b00feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.412+0000 7f31b5625700 1 -- 172.21.15.106:0/187546493 shutdown_connections 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.412+0000 7f31b5625700 1 -- 172.21.15.106:0/187546493 wait complete. 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31b5625700 1 Processor -- start 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31b5625700 1 -- start start 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31b5625700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b0101af0 0x7f31b019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31b5625700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b01024a0 0x7f31b019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31b5625700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b0106410 0x7f31b0198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31b5625700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b01113f0 con 0x7f31b0101af0 2024-03-28T02:10:16.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31b5625700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b0111270 con 0x7f31b01024a0 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31b5625700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b0111570 con 0x7f31b0106410 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31aeffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b0101af0 0x7f31b019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31aeffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b0101af0 0x7f31b019e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51946/0 (socket says 172.21.15.106:51946) 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31aeffd700 1 -- 172.21.15.106:0/3107289351 learned_addr learned my addr 172.21.15.106:0/3107289351 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31ae7fc700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b01024a0 0x7f31b019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.413+0000 7f31af7fe700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b0106410 0x7f31b0198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31aeffd700 1 -- 172.21.15.106:0/3107289351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b0106410 msgr2=0x7f31b0198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31aeffd700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b0106410 0x7f31b0198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31aeffd700 1 -- 172.21.15.106:0/3107289351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b01024a0 msgr2=0x7f31b019eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31aeffd700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b01024a0 0x7f31b019eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31aeffd700 1 -- 172.21.15.106:0/3107289351 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0199000 con 0x7f31b0101af0 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31af7fe700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b0106410 0x7f31b0198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31ae7fc700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b01024a0 0x7f31b019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31aeffd700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b0101af0 0x7f31b019e650 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f31a0012dd0 tx=0x7f31a000c5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:16.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f3197fff700 1 -- 172.21.15.106:0/3107289351 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a0016030 con 0x7f31b0101af0 2024-03-28T02:10:16.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b01992f0 con 0x7f31b0101af0 2024-03-28T02:10:16.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f3197fff700 1 -- 172.21.15.106:0/3107289351 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31a00125b0 con 0x7f31b0101af0 2024-03-28T02:10:16.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f3197fff700 1 -- 172.21.15.106:0/3107289351 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a0005280 con 0x7f31b0101af0 2024-03-28T02:10:16.053 INFO:teuthology.orchestra.run.smithi106.stdout:343597383711 2024-03-28T02:10:16.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.414+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b0107fb0 con 0x7f31b0101af0 2024-03-28T02:10:16.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.415+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b0068e50 con 0x7f31b0101af0 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.417+0000 7f3197fff700 1 -- 172.21.15.106:0/3107289351 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f31a0019070 con 0x7f31b0101af0 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.418+0000 7f3197fff700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31980a2890 0x7f31980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.418+0000 7f3197fff700 1 -- 172.21.15.106:0/3107289351 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f31a00cecf0 con 0x7f31b0101af0 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.418+0000 7f31ae7fc700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31980a2890 0x7f31980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.418+0000 7f31ae7fc700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31980a2890 0x7f31980a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f31a4000c00 tx=0x7f31a4024040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.422+0000 7f3197fff700 1 -- 172.21.15.106:0/3107289351 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f31a0019360 con 0x7f31b0101af0 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.632+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f31b0102880 con 0x7f31b0101af0 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.632+0000 7f3197fff700 1 -- 172.21.15.106:0/3107289351 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f31a009ba50 con 0x7f31b0101af0 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31980a2890 msgr2=0x7f31980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:16.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31980a2890 0x7f31980a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f31a4000c00 tx=0x7f31a4024040 comp rx=0 tx=0).stop 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b0101af0 msgr2=0x7f31b019e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b0101af0 0x7f31b019e650 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f31a0012dd0 tx=0x7f31a000c5b0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 shutdown_connections 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b0106410 0x7f31b0198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31980a2890 0x7f31980a4d50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31b0101af0 0x7f31b019e650 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 --2- 172.21.15.106:0/3107289351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b01024a0 0x7f31b019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 >> 172.21.15.106:0/3107289351 conn(0x7f31b0077690 msgr2=0x7f31b0107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 shutdown_connections 2024-03-28T02:10:16.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:15.634+0000 7f31b5625700 1 -- 172.21.15.106:0/3107289351 wait complete. 2024-03-28T02:10:16.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:16 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3107289351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T02:10:16.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:16 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3508205292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T02:10:16.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:16 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3107289351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T02:10:16.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:16 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3508205292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T02:10:16.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:16 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3107289351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T02:10:16.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:16 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3508205292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T02:10:17.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:17 smithi106 ceph-mon[31965]: pgmap v338: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:17 smithi178 ceph-mon[36521]: pgmap v338: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:17 smithi153 ceph-mon[36228]: pgmap v338: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:18.221 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411401 got 77309411404 for osd.2 2024-03-28T02:10:18.222 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:18.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 -- 172.21.15.106:0/3755174306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1082e0 msgr2=0x7f6a6c1086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 --2- 172.21.15.106:0/3755174306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1082e0 0x7f6a6c1086c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6a600099c0 tx=0x7f6a60009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 -- 172.21.15.106:0/3755174306 shutdown_connections 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 --2- 172.21.15.106:0/3755174306 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a6c1010a0 0x7f6a6c10f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 --2- 172.21.15.106:0/3755174306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1082e0 0x7f6a6c1086c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 --2- 172.21.15.106:0/3755174306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c1006e0 0x7f6a6c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 -- 172.21.15.106:0/3755174306 >> 172.21.15.106:0/3755174306 conn(0x7f6a6c077660 msgr2=0x7f6a6c0fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 -- 172.21.15.106:0/3755174306 shutdown_connections 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 -- 172.21.15.106:0/3755174306 wait complete. 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 Processor -- start 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.421+0000 7f6a72f26700 1 -- start start 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a72f26700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a6c1006e0 0x7f6a6c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:18.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a72f26700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1010a0 0x7f6a6c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a72f26700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c1082e0 0x7f6a6c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a72f26700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c112570 con 0x7f6a6c1010a0 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a72f26700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c1123f0 con 0x7f6a6c1082e0 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a72f26700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c1126f0 con 0x7f6a6c1006e0 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a6bfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1010a0 0x7f6a6c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a6bfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1010a0 0x7f6a6c19ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52674/0 (socket says 172.21.15.106:52674) 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a6bfff700 1 -- 172.21.15.106:0/2923195569 learned_addr learned my addr 172.21.15.106:0/2923195569 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a70cc2700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a6c1006e0 0x7f6a6c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a714c3700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c1082e0 0x7f6a6c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a6bfff700 1 -- 172.21.15.106:0/2923195569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a6c1006e0 msgr2=0x7f6a6c19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a6bfff700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a6c1006e0 0x7f6a6c19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a6bfff700 1 -- 172.21.15.106:0/2923195569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c1082e0 msgr2=0x7f6a6c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a6bfff700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c1082e0 0x7f6a6c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a6bfff700 1 -- 172.21.15.106:0/2923195569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c198fd0 con 0x7f6a6c1010a0 2024-03-28T02:10:18.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a6bfff700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1010a0 0x7f6a6c19ebb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e9a0 tx=0x7f6a5800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:18.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a69ffb700 1 -- 172.21.15.106:0/2923195569 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5800ca10 con 0x7f6a6c1010a0 2024-03-28T02:10:18.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a69ffb700 1 -- 172.21.15.106:0/2923195569 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a5800cbb0 con 0x7f6a6c1010a0 2024-03-28T02:10:18.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6c1992c0 con 0x7f6a6c1010a0 2024-03-28T02:10:18.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.422+0000 7f6a69ffb700 1 -- 172.21.15.106:0/2923195569 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58018680 con 0x7f6a6c1010a0 2024-03-28T02:10:18.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.423+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6c1a54d0 con 0x7f6a6c1010a0 2024-03-28T02:10:18.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.424+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6c103a00 con 0x7f6a6c1010a0 2024-03-28T02:10:18.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.425+0000 7f6a69ffb700 1 -- 172.21.15.106:0/2923195569 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6a58010430 con 0x7f6a6c1010a0 2024-03-28T02:10:18.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.425+0000 7f6a69ffb700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6a5c0a2910 0x7f6a5c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:18.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.425+0000 7f6a70cc2700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6a5c0a2910 0x7f6a5c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:18.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.425+0000 7f6a69ffb700 1 -- 172.21.15.106:0/2923195569 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f6a580cf3f0 con 0x7f6a6c1010a0 2024-03-28T02:10:18.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.426+0000 7f6a70cc2700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6a5c0a2910 0x7f6a5c0a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a60005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:18.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.429+0000 7f6a69ffb700 1 -- 172.21.15.106:0/2923195569 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6a5809c150 con 0x7f6a6c1010a0 2024-03-28T02:10:18.626 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.632+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6a6c02da30 con 0x7f6a6c1010a0 2024-03-28T02:10:18.626 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.632+0000 7f6a69ffb700 1 -- 172.21.15.106:0/2923195569 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6a5809ba50 con 0x7f6a6c1010a0 2024-03-28T02:10:18.626 INFO:teuthology.orchestra.run.smithi106.stdout:197568495674 2024-03-28T02:10:18.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.634+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6a5c0a2910 msgr2=0x7f6a5c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:18.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6a5c0a2910 0x7f6a5c0a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a60005fb0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1010a0 msgr2=0x7f6a6c19ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:18.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1010a0 0x7f6a6c19ebb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e9a0 tx=0x7f6a5800ecb0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 shutdown_connections 2024-03-28T02:10:18.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c1082e0 0x7f6a6c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a6c1006e0 0x7f6a6c19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6a5c0a2910 0x7f6a5c0a4dd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 --2- 172.21.15.106:0/2923195569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c1010a0 0x7f6a6c19ebb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:18.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 >> 172.21.15.106:0/2923195569 conn(0x7f6a6c077660 msgr2=0x7f6a6c105100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:18.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 shutdown_connections 2024-03-28T02:10:18.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:18.635+0000 7f6a72f26700 1 -- 172.21.15.106:0/2923195569 wait complete. 2024-03-28T02:10:18.934 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757304 got 163208757310 for osd.5 2024-03-28T02:10:18.934 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:19.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:19 smithi106 ceph-mon[31965]: pgmap v339: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T02:10:19.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:19 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2923195569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T02:10:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:19 smithi178 ceph-mon[36521]: pgmap v339: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T02:10:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:19 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2923195569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T02:10:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:19 smithi153 ceph-mon[36228]: pgmap v339: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T02:10:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:19 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2923195569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.144+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3203076208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc41023d0 msgr2=0x7f8bc41027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.144+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3203076208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc41023d0 0x7f8bc41027b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8bc000b0d0 tx=0x7f8bc000b3e0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.144+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3203076208 shutdown_connections 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.144+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3203076208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc41074b0 0x7f8bc410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.144+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3203076208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc41023d0 0x7f8bc41027b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.144+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3203076208 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bc4102d80 0x7f8bc4106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.144+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3203076208 >> 172.21.15.106:0/3203076208 conn(0x7f8bc40fdc40 msgr2=0x7f8bc4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.144+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3203076208 shutdown_connections 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.144+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3203076208 wait complete. 2024-03-28T02:10:20.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bcb23f700 1 Processor -- start 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bcb23f700 1 -- start start 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bcb23f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc41023d0 0x7f8bc4074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bcb23f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc4102d80 0x7f8bc40749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bcb23f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bc41074b0 0x7f8bc4074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bcb23f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc41124b0 con 0x7f8bc4102d80 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bcb23f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc4112330 con 0x7f8bc41023d0 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bcb23f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc4112630 con 0x7f8bc41074b0 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bbbfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc4102d80 0x7f8bc40749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bbbfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc4102d80 0x7f8bc40749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52694/0 (socket says 172.21.15.106:52694) 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bbbfff700 1 -- 172.21.15.106:0/3923464297 learned_addr learned my addr 172.21.15.106:0/3923464297 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bc8fdb700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc41023d0 0x7f8bc4074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bc97dc700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bc41074b0 0x7f8bc4074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bbbfff700 1 -- 172.21.15.106:0/3923464297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bc41074b0 msgr2=0x7f8bc4074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:20.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bbbfff700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bc41074b0 0x7f8bc4074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bbbfff700 1 -- 172.21.15.106:0/3923464297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc41023d0 msgr2=0x7f8bc4074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bbbfff700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc41023d0 0x7f8bc4074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.145+0000 7f8bbbfff700 1 -- 172.21.15.106:0/3923464297 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc41a6e80 con 0x7f8bc4102d80 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.146+0000 7f8bbbfff700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc4102d80 0x7f8bc40749d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400e9f0 tx=0x7f8bb400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.146+0000 7f8bb9ffb700 1 -- 172.21.15.106:0/3923464297 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4004a30 con 0x7f8bc4102d80 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.146+0000 7f8bb9ffb700 1 -- 172.21.15.106:0/3923464297 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8bb400cca0 con 0x7f8bc4102d80 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.146+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc41a7170 con 0x7f8bc4102d80 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.146+0000 7f8bb9ffb700 1 -- 172.21.15.106:0/3923464297 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb40056c0 con 0x7f8bc4102d80 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.146+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc41a7660 con 0x7f8bc4102d80 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.147+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc4068e50 con 0x7f8bc4102d80 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.148+0000 7f8bb9ffb700 1 -- 172.21.15.106:0/3923464297 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8bb400c7c0 con 0x7f8bc4102d80 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.148+0000 7f8bb9ffb700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8bac0a2890 0x7f8bac0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.149+0000 7f8bc8fdb700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8bac0a2890 0x7f8bac0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:20.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.149+0000 7f8bb9ffb700 1 -- 172.21.15.106:0/3923464297 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f8bb40cf170 con 0x7f8bc4102d80 2024-03-28T02:10:20.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.149+0000 7f8bc8fdb700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8bac0a2890 0x7f8bac0a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009fd0 tx=0x7f8bc000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:20.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.153+0000 7f8bb9ffb700 1 -- 172.21.15.106:0/3923464297 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8bb409bed0 con 0x7f8bc4102d80 2024-03-28T02:10:20.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.351+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8bc4075970 con 0x7f8bc4102d80 2024-03-28T02:10:20.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.351+0000 7f8bb9ffb700 1 -- 172.21.15.106:0/3923464297 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8bb409b7d0 con 0x7f8bc4102d80 2024-03-28T02:10:20.345 INFO:teuthology.orchestra.run.smithi106.stdout:219043332148 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8bac0a2890 msgr2=0x7f8bac0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8bac0a2890 0x7f8bac0a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009fd0 tx=0x7f8bc000b040 comp rx=0 tx=0).stop 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc4102d80 msgr2=0x7f8bc40749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc4102d80 0x7f8bc40749d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400e9f0 tx=0x7f8bb400ed00 comp rx=0 tx=0).stop 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 shutdown_connections 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bc41074b0 0x7f8bc4074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc41023d0 0x7f8bc4074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8bac0a2890 0x7f8bac0a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 --2- 172.21.15.106:0/3923464297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8bc4102d80 0x7f8bc40749d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 >> 172.21.15.106:0/3923464297 conn(0x7f8bc40fdc40 msgr2=0x7f8bc40fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 shutdown_connections 2024-03-28T02:10:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:20.353+0000 7f8bcb23f700 1 -- 172.21.15.106:0/3923464297 wait complete. 2024-03-28T02:10:20.747 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:20 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3923464297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T02:10:20.774 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808869 got 287762808874 for osd.9 2024-03-28T02:10:20.774 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:20.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:20 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3923464297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T02:10:20.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:20 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3923464297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T02:10:21.196 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495668 got 197568495674 for osd.6 2024-03-28T02:10:21.196 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:21.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:21 smithi106 ceph-mon[31965]: pgmap v340: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:21 smithi178 ceph-mon[36521]: pgmap v340: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:21.929 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332143 got 219043332148 for osd.7 2024-03-28T02:10:21.929 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:21 smithi153 ceph-mon[36228]: pgmap v340: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:22.399 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383706 got 343597383711 for osd.11 2024-03-28T02:10:22.400 DEBUG:teuthology.parallel:result is None 2024-03-28T02:10:22.400 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T02:10:22.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T02:10:22.684 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:23.813 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:23 smithi106 ceph-mon[31965]: pgmap v341: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:23.814 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:10:23.814 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:10:23.814 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:10:23.814 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:10:23.814 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:23 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:10:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:23 smithi178 ceph-mon[36521]: pgmap v341: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:10:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:10:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:10:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:10:23.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:23 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:10:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:23 smithi153 ceph-mon[36228]: pgmap v341: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:10:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:10:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:10:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:10:23.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:23 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' 2024-03-28T02:10:24.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.500+0000 7f10e8d38700 1 -- 172.21.15.106:0/612641218 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 msgr2=0x7f10e410df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:24.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.500+0000 7f10e8d38700 1 --2- 172.21.15.106:0/612641218 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 0x7f10e410df10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00b0a0 tx=0x7f10dc00b3b0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.500+0000 7f10e8d38700 1 -- 172.21.15.106:0/612641218 shutdown_connections 2024-03-28T02:10:24.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.500+0000 7f10e8d38700 1 --2- 172.21.15.106:0/612641218 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 0x7f10e410df10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.500+0000 7f10e8d38700 1 --2- 172.21.15.106:0/612641218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10e41070d0 0x7f10e41074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.500+0000 7f10e8d38700 1 --2- 172.21.15.106:0/612641218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10e4101090 0x7f10e41014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.500+0000 7f10e8d38700 1 -- 172.21.15.106:0/612641218 >> 172.21.15.106:0/612641218 conn(0x7f10e4077fc0 msgr2=0x7f10e40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:24.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.500+0000 7f10e8d38700 1 -- 172.21.15.106:0/612641218 shutdown_connections 2024-03-28T02:10:24.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.500+0000 7f10e8d38700 1 -- 172.21.15.106:0/612641218 wait complete. 2024-03-28T02:10:24.495 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e8d38700 1 Processor -- start 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e8d38700 1 -- start start 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e8d38700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10e4101090 0x7f10e419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e8d38700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 0x7f10e419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e8d38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10e41070d0 0x7f10e4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e8d38700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e4111350 con 0x7f10e4101a30 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e8d38700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e41111d0 con 0x7f10e41070d0 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e8d38700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e41114d0 con 0x7f10e4101090 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e1d9b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 0x7f10e419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e1d9b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 0x7f10e419ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52714/0 (socket says 172.21.15.106:52714) 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e1d9b700 1 -- 172.21.15.106:0/3018582368 learned_addr learned my addr 172.21.15.106:0/3018582368 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e259c700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10e4101090 0x7f10e419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e2d9d700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10e41070d0 0x7f10e4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e1d9b700 1 -- 172.21.15.106:0/3018582368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10e4101090 msgr2=0x7f10e419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e1d9b700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10e4101090 0x7f10e419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e1d9b700 1 -- 172.21.15.106:0/3018582368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10e41070d0 msgr2=0x7f10e4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e1d9b700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10e41070d0 0x7f10e4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e1d9b700 1 -- 172.21.15.106:0/3018582368 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e4199040 con 0x7f10e4101a30 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.501+0000 7f10e259c700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10e4101090 0x7f10e419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.502+0000 7f10e1d9b700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 0x7f10e419ebd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f10d800e9c0 tx=0x7f10d800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.502+0000 7f10d37fe700 1 -- 172.21.15.106:0/3018582368 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d800cbd0 con 0x7f10e4101a30 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.502+0000 7f10d37fe700 1 -- 172.21.15.106:0/3018582368 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10d800ee60 con 0x7f10e4101a30 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.502+0000 7f10d37fe700 1 -- 172.21.15.106:0/3018582368 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8005630 con 0x7f10e4101a30 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.502+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e4199330 con 0x7f10e4101a30 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.502+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e41a5610 con 0x7f10e4101a30 2024-03-28T02:10:24.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.503+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e4102780 con 0x7f10e4101a30 2024-03-28T02:10:24.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.504+0000 7f10d37fe700 1 -- 172.21.15.106:0/3018582368 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f10d8004750 con 0x7f10e4101a30 2024-03-28T02:10:24.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.504+0000 7f10d37fe700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f10cc0a2890 0x7f10cc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:24.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.505+0000 7f10e259c700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f10cc0a2890 0x7f10cc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:24.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.505+0000 7f10d37fe700 1 -- 172.21.15.106:0/3018582368 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f10d80cf020 con 0x7f10e4101a30 2024-03-28T02:10:24.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.505+0000 7f10e259c700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f10cc0a2890 0x7f10cc0a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009fd0 tx=0x7f10d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:24.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.509+0000 7f10d37fe700 1 -- 172.21.15.106:0/3018582368 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f10d809bd80 con 0x7f10e4101a30 2024-03-28T02:10:24.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.701+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10e4104e80 con 0x7f10cc0a2890 2024-03-28T02:10:24.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.713+0000 7f10d37fe700 1 -- 172.21.15.106:0/3018582368 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340814 (secure 0 0 0) 0x7f10e4104e80 con 0x7f10cc0a2890 2024-03-28T02:10:24.711 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:24.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f10cc0a2890 msgr2=0x7f10cc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:24.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f10cc0a2890 0x7f10cc0a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009fd0 tx=0x7f10d4009260 comp rx=0 tx=0).stop 2024-03-28T02:10:24.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 msgr2=0x7f10e419ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:24.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 0x7f10e419ebd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f10d800e9c0 tx=0x7f10d800ecd0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 shutdown_connections 2024-03-28T02:10:24.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10e41070d0 0x7f10e4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10e4101090 0x7f10e419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f10cc0a2890 0x7f10cc0a4d50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 --2- 172.21.15.106:0/3018582368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e4101a30 0x7f10e419ebd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:24.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 >> 172.21.15.106:0/3018582368 conn(0x7f10e4077fc0 msgr2=0x7f10e4100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:24.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 shutdown_connections 2024-03-28T02:10:24.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:24.716+0000 7f10e8d38700 1 -- 172.21.15.106:0/3018582368 wait complete. 2024-03-28T02:10:24.714 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-03-28T02:10:25.508 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-03-28T02:10:24.377651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664549,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":902,"num_read_kb":586,"num_write":429,"num_write_kb":3086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1181033,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":854,"ondisk_log_size":854,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378112,"kb_used_data":12320,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563712,"statfs":{"total":1151940427776,"available":1151553241088,"internally_reserved":0,"allocated":12615680,"data_stored":7978697,"data_compressed":355965,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":71,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003244"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.397218+0000","last_change":"2024-03-28T02:08:49.268140+0000","last_active":"2024-03-28T02:10:12.397218+0000","last_peered":"2024-03-28T02:10:12.397218+0000","last_clean":"2024-03-28T02:10:12.397218+0000","last_became_active":"2024-03-28T02:08:49.267695+0000","last_became_peered":"2024-03-28T02:08:49.267695+0000","last_unstale":"2024-03-28T02:10:12.397218+0000","last_undegraded":"2024-03-28T02:10:12.397218+0000","last_fullsized":"2024-03-28T02:10:12.397218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:27:09.768237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.385752+0000","last_change":"2024-03-28T02:08:49.259914+0000","last_active":"2024-03-28T02:10:12.385752+0000","last_peered":"2024-03-28T02:10:12.385752+0000","last_clean":"2024-03-28T02:10:12.385752+0000","last_became_active":"2024-03-28T02:08:49.259579+0000","last_became_peered":"2024-03-28T02:08:49.259579+0000","last_unstale":"2024-03-28T02:10:12.385752+0000","last_undegraded":"2024-03-28T02:10:12.385752+0000","last_fullsized":"2024-03-28T02:10:12.385752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:04:02.664429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402092+0000","last_change":"2024-03-28T02:08:49.268234+0000","last_active":"2024-03-28T02:10:12.402092+0000","last_peered":"2024-03-28T02:10:12.402092+0000","last_clean":"2024-03-28T02:10:12.402092+0000","last_became_active":"2024-03-28T02:08:49.267813+0000","last_became_peered":"2024-03-28T02:08:49.267813+0000","last_unstale":"2024-03-28T02:10:12.402092+0000","last_undegraded":"2024-03-28T02:10:12.402092+0000","last_fullsized":"2024-03-28T02:10:12.402092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:46:21.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.395926+0000","last_change":"2024-03-28T02:08:49.268397+0000","last_active":"2024-03-28T02:10:12.395926+0000","last_peered":"2024-03-28T02:10:12.395926+0000","last_clean":"2024-03-28T02:10:12.395926+0000","last_became_active":"2024-03-28T02:08:49.268059+0000","last_became_peered":"2024-03-28T02:08:49.268059+0000","last_unstale":"2024-03-28T02:10:12.395926+0000","last_undegraded":"2024-03-28T02:10:12.395926+0000","last_fullsized":"2024-03-28T02:10:12.395926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:45:51.951306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.385728+0000","last_change":"2024-03-28T02:08:49.257827+0000","last_active":"2024-03-28T02:10:12.385728+0000","last_peered":"2024-03-28T02:10:12.385728+0000","last_clean":"2024-03-28T02:10:12.385728+0000","last_became_active":"2024-03-28T02:08:49.256707+0000","last_became_peered":"2024-03-28T02:08:49.256707+0000","last_unstale":"2024-03-28T02:10:12.385728+0000","last_undegraded":"2024-03-28T02:10:12.385728+0000","last_fullsized":"2024-03-28T02:10:12.385728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:23:13.617900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.384334+0000","last_change":"2024-03-28T02:08:49.262545+0000","last_active":"2024-03-28T02:10:12.384334+0000","last_peered":"2024-03-28T02:10:12.384334+0000","last_clean":"2024-03-28T02:10:12.384334+0000","last_became_active":"2024-03-28T02:08:49.262218+0000","last_became_peered":"2024-03-28T02:08:49.262218+0000","last_unstale":"2024-03-28T02:10:12.384334+0000","last_undegraded":"2024-03-28T02:10:12.384334+0000","last_fullsized":"2024-03-28T02:10:12.384334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:55:20.418608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.386346+0000","last_change":"2024-03-28T02:08:49.260072+0000","last_active":"2024-03-28T02:10:12.386346+0000","last_peered":"2024-03-28T02:10:12.386346+0000","last_clean":"2024-03-28T02:10:12.386346+0000","last_became_active":"2024-03-28T02:08:49.259724+0000","last_became_peered":"2024-03-28T02:08:49.259724+0000","last_unstale":"2024-03-28T02:10:12.386346+0000","last_undegraded":"2024-03-28T02:10:12.386346+0000","last_fullsized":"2024-03-28T02:10:12.386346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:05.445467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.387161+0000","last_change":"2024-03-28T02:08:49.262547+0000","last_active":"2024-03-28T02:10:12.387161+0000","last_peered":"2024-03-28T02:10:12.387161+0000","last_clean":"2024-03-28T02:10:12.387161+0000","last_became_active":"2024-03-28T02:08:49.261836+0000","last_became_peered":"2024-03-28T02:08:49.261836+0000","last_unstale":"2024-03-28T02:10:12.387161+0000","last_undegraded":"2024-03-28T02:10:12.387161+0000","last_fullsized":"2024-03-28T02:10:12.387161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:55:22.201524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.403467+0000","last_change":"2024-03-28T02:08:49.262725+0000","last_active":"2024-03-28T02:10:12.403467+0000","last_peered":"2024-03-28T02:10:12.403467+0000","last_clean":"2024-03-28T02:10:12.403467+0000","last_became_active":"2024-03-28T02:08:49.262319+0000","last_became_peered":"2024-03-28T02:08:49.262319+0000","last_unstale":"2024-03-28T02:10:12.403467+0000","last_undegraded":"2024-03-28T02:10:12.403467+0000","last_fullsized":"2024-03-28T02:10:12.403467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:56:24.904009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.389220+0000","last_change":"2024-03-28T02:08:49.262194+0000","last_active":"2024-03-28T02:10:12.389220+0000","last_peered":"2024-03-28T02:10:12.389220+0000","last_clean":"2024-03-28T02:10:12.389220+0000","last_became_active":"2024-03-28T02:08:49.261887+0000","last_became_peered":"2024-03-28T02:08:49.261887+0000","last_unstale":"2024-03-28T02:10:12.389220+0000","last_undegraded":"2024-03-28T02:10:12.389220+0000","last_fullsized":"2024-03-28T02:10:12.389220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:52:47.049430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402139+0000","last_change":"2024-03-28T02:08:49.269222+0000","last_active":"2024-03-28T02:10:12.402139+0000","last_peered":"2024-03-28T02:10:12.402139+0000","last_clean":"2024-03-28T02:10:12.402139+0000","last_became_active":"2024-03-28T02:08:49.268647+0000","last_became_peered":"2024-03-28T02:08:49.268647+0000","last_unstale":"2024-03-28T02:10:12.402139+0000","last_undegraded":"2024-03-28T02:10:12.402139+0000","last_fullsized":"2024-03-28T02:10:12.402139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:10:01.108158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.389425+0000","last_change":"2024-03-28T02:08:49.269868+0000","last_active":"2024-03-28T02:10:12.389425+0000","last_peered":"2024-03-28T02:10:12.389425+0000","last_clean":"2024-03-28T02:10:12.389425+0000","last_became_active":"2024-03-28T02:08:49.269035+0000","last_became_peered":"2024-03-28T02:08:49.269035+0000","last_unstale":"2024-03-28T02:10:12.389425+0000","last_undegraded":"2024-03-28T02:10:12.389425+0000","last_fullsized":"2024-03-28T02:10:12.389425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:31.153970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388726+0000","last_change":"2024-03-28T02:08:49.254162+0000","last_active":"2024-03-28T02:10:12.388726+0000","last_peered":"2024-03-28T02:10:12.388726+0000","last_clean":"2024-03-28T02:10:12.388726+0000","last_became_active":"2024-03-28T02:08:49.253756+0000","last_became_peered":"2024-03-28T02:08:49.253756+0000","last_unstale":"2024-03-28T02:10:12.388726+0000","last_undegraded":"2024-03-28T02:10:12.388726+0000","last_fullsized":"2024-03-28T02:10:12.388726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:22:20.791809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388266+0000","last_change":"2024-03-28T02:08:49.257940+0000","last_active":"2024-03-28T02:10:12.388266+0000","last_peered":"2024-03-28T02:10:12.388266+0000","last_clean":"2024-03-28T02:10:12.388266+0000","last_became_active":"2024-03-28T02:08:49.257039+0000","last_became_peered":"2024-03-28T02:08:49.257039+0000","last_unstale":"2024-03-28T02:10:12.388266+0000","last_undegraded":"2024-03-28T02:10:12.388266+0000","last_fullsized":"2024-03-28T02:10:12.388266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:36:10.701760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.393556+0000","last_change":"2024-03-28T02:08:49.267502+0000","last_active":"2024-03-28T02:10:12.393556+0000","last_peered":"2024-03-28T02:10:12.393556+0000","last_clean":"2024-03-28T02:10:12.393556+0000","last_became_active":"2024-03-28T02:08:49.265447+0000","last_became_peered":"2024-03-28T02:08:49.265447+0000","last_unstale":"2024-03-28T02:10:12.393556+0000","last_undegraded":"2024-03-28T02:10:12.393556+0000","last_fullsized":"2024-03-28T02:10:12.393556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:47:44.891440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.403441+0000","last_change":"2024-03-28T02:08:49.267702+0000","last_active":"2024-03-28T02:10:12.403441+0000","last_peered":"2024-03-28T02:10:12.403441+0000","last_clean":"2024-03-28T02:10:12.403441+0000","last_became_active":"2024-03-28T02:08:49.267381+0000","last_became_peered":"2024-03-28T02:08:49.267381+0000","last_unstale":"2024-03-28T02:10:12.403441+0000","last_undegraded":"2024-03-28T02:10:12.403441+0000","last_fullsized":"2024-03-28T02:10:12.403441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:55:59.904141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.397366+0000","last_change":"2024-03-28T02:08:49.268240+0000","last_active":"2024-03-28T02:10:12.397366+0000","last_peered":"2024-03-28T02:10:12.397366+0000","last_clean":"2024-03-28T02:10:12.397366+0000","last_became_active":"2024-03-28T02:08:49.267884+0000","last_became_peered":"2024-03-28T02:08:49.267884+0000","last_unstale":"2024-03-28T02:10:12.397366+0000","last_undegraded":"2024-03-28T02:10:12.397366+0000","last_fullsized":"2024-03-28T02:10:12.397366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:34:23.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.384295+0000","last_change":"2024-03-28T02:08:49.262585+0000","last_active":"2024-03-28T02:10:12.384295+0000","last_peered":"2024-03-28T02:10:12.384295+0000","last_clean":"2024-03-28T02:10:12.384295+0000","last_became_active":"2024-03-28T02:08:49.262217+0000","last_became_peered":"2024-03-28T02:08:49.262217+0000","last_unstale":"2024-03-28T02:10:12.384295+0000","last_undegraded":"2024-03-28T02:10:12.384295+0000","last_fullsized":"2024-03-28T02:10:12.384295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:10:05.714733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.392792+0000","last_change":"2024-03-28T02:08:49.261550+0000","last_active":"2024-03-28T02:10:12.392792+0000","last_peered":"2024-03-28T02:10:12.392792+0000","last_clean":"2024-03-28T02:10:12.392792+0000","last_became_active":"2024-03-28T02:08:49.261196+0000","last_became_peered":"2024-03-28T02:08:49.261196+0000","last_unstale":"2024-03-28T02:10:12.392792+0000","last_undegraded":"2024-03-28T02:10:12.392792+0000","last_fullsized":"2024-03-28T02:10:12.392792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:52:52.972048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457157+0000","last_change":"2024-03-28T02:08:53.491877+0000","last_active":"2024-03-28T02:09:43.457157+0000","last_peered":"2024-03-28T02:09:43.457157+0000","last_clean":"2024-03-28T02:09:43.457157+0000","last_became_active":"2024-03-28T02:08:53.491584+0000","last_became_peered":"2024-03-28T02:08:53.491584+0000","last_unstale":"2024-03-28T02:09:43.457157+0000","last_undegraded":"2024-03-28T02:09:43.457157+0000","last_fullsized":"2024-03-28T02:09:43.457157+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:47:41.778839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:22.380349+0000","last_change":"2024-03-28T02:08:48.448277+0000","last_active":"2024-03-28T02:10:22.380349+0000","last_peered":"2024-03-28T02:10:22.380349+0000","last_clean":"2024-03-28T02:10:22.380349+0000","last_became_active":"2024-03-28T02:08:48.447910+0000","last_became_peered":"2024-03-28T02:08:48.447910+0000","last_unstale":"2024-03-28T02:10:22.380349+0000","last_undegraded":"2024-03-28T02:10:22.380349+0000","last_fullsized":"2024-03-28T02:10:22.380349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:13:34.241952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390546+0000","last_change":"2024-03-28T02:08:49.258840+0000","last_active":"2024-03-28T02:10:12.390546+0000","last_peered":"2024-03-28T02:10:12.390546+0000","last_clean":"2024-03-28T02:10:12.390546+0000","last_became_active":"2024-03-28T02:08:49.258259+0000","last_became_peered":"2024-03-28T02:08:49.258259+0000","last_unstale":"2024-03-28T02:10:12.390546+0000","last_undegraded":"2024-03-28T02:10:12.390546+0000","last_fullsized":"2024-03-28T02:10:12.390546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:25:31.170413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459442+0000","last_change":"2024-03-28T02:08:53.496476+0000","last_active":"2024-03-28T02:09:43.459442+0000","last_peered":"2024-03-28T02:09:43.459442+0000","last_clean":"2024-03-28T02:09:43.459442+0000","last_became_active":"2024-03-28T02:08:53.496034+0000","last_became_peered":"2024-03-28T02:08:53.496034+0000","last_unstale":"2024-03-28T02:09:43.459442+0000","last_undegraded":"2024-03-28T02:09:43.459442+0000","last_fullsized":"2024-03-28T02:09:43.459442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:49:00.550877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.464780+0000","last_change":"2024-03-28T02:08:48.457960+0000","last_active":"2024-03-28T02:09:43.464780+0000","last_peered":"2024-03-28T02:09:43.464780+0000","last_clean":"2024-03-28T02:09:43.464780+0000","last_became_active":"2024-03-28T02:08:48.455528+0000","last_became_peered":"2024-03-28T02:08:48.455528+0000","last_unstale":"2024-03-28T02:09:43.464780+0000","last_undegraded":"2024-03-28T02:09:43.464780+0000","last_fullsized":"2024-03-28T02:09:43.464780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:40:37.830166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402071+0000","last_change":"2024-03-28T02:08:49.268390+0000","last_active":"2024-03-28T02:10:12.402071+0000","last_peered":"2024-03-28T02:10:12.402071+0000","last_clean":"2024-03-28T02:10:12.402071+0000","last_became_active":"2024-03-28T02:08:49.268025+0000","last_became_peered":"2024-03-28T02:08:49.268025+0000","last_unstale":"2024-03-28T02:10:12.402071+0000","last_undegraded":"2024-03-28T02:10:12.402071+0000","last_fullsized":"2024-03-28T02:10:12.402071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:40:07.752553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.465487+0000","last_change":"2024-03-28T02:09:43.465487+0000","last_active":"2024-03-28T02:09:43.465487+0000","last_peered":"2024-03-28T02:09:43.465487+0000","last_clean":"2024-03-28T02:09:43.465487+0000","last_became_active":"2024-03-28T02:09:43.465015+0000","last_became_peered":"2024-03-28T02:09:43.465015+0000","last_unstale":"2024-03-28T02:09:43.465487+0000","last_undegraded":"2024-03-28T02:09:43.465487+0000","last_fullsized":"2024-03-28T02:09:43.465487+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:48:58.978284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,8],"acting":[4,3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461836+0000","last_change":"2024-03-28T02:08:48.459667+0000","last_active":"2024-03-28T02:09:43.461836+0000","last_peered":"2024-03-28T02:09:43.461836+0000","last_clean":"2024-03-28T02:09:43.461836+0000","last_became_active":"2024-03-28T02:08:48.459310+0000","last_became_peered":"2024-03-28T02:08:48.459310+0000","last_unstale":"2024-03-28T02:09:43.461836+0000","last_undegraded":"2024-03-28T02:09:43.461836+0000","last_fullsized":"2024-03-28T02:09:43.461836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:51:07.351151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.403849+0000","last_change":"2024-03-28T02:08:49.254576+0000","last_active":"2024-03-28T02:10:12.403849+0000","last_peered":"2024-03-28T02:10:12.403849+0000","last_clean":"2024-03-28T02:10:12.403849+0000","last_became_active":"2024-03-28T02:08:49.254213+0000","last_became_peered":"2024-03-28T02:08:49.254213+0000","last_unstale":"2024-03-28T02:10:12.403849+0000","last_undegraded":"2024-03-28T02:10:12.403849+0000","last_fullsized":"2024-03-28T02:10:12.403849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.630801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459713+0000","last_change":"2024-03-28T02:08:53.493624+0000","last_active":"2024-03-28T02:09:43.459713+0000","last_peered":"2024-03-28T02:09:43.459713+0000","last_clean":"2024-03-28T02:09:43.459713+0000","last_became_active":"2024-03-28T02:08:53.493195+0000","last_became_peered":"2024-03-28T02:08:53.493195+0000","last_unstale":"2024-03-28T02:09:43.459713+0000","last_undegraded":"2024-03-28T02:09:43.459713+0000","last_fullsized":"2024-03-28T02:09:43.459713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:38:25.497464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.462020+0000","last_change":"2024-03-28T02:08:48.454008+0000","last_active":"2024-03-28T02:09:43.462020+0000","last_peered":"2024-03-28T02:09:43.462020+0000","last_clean":"2024-03-28T02:09:43.462020+0000","last_became_active":"2024-03-28T02:08:48.450674+0000","last_became_peered":"2024-03-28T02:08:48.450674+0000","last_unstale":"2024-03-28T02:09:43.462020+0000","last_undegraded":"2024-03-28T02:09:43.462020+0000","last_fullsized":"2024-03-28T02:09:43.462020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:08:12.902348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402185+0000","last_change":"2024-03-28T02:08:49.266552+0000","last_active":"2024-03-28T02:10:12.402185+0000","last_peered":"2024-03-28T02:10:12.402185+0000","last_clean":"2024-03-28T02:10:12.402185+0000","last_became_active":"2024-03-28T02:08:49.265161+0000","last_became_peered":"2024-03-28T02:08:49.265161+0000","last_unstale":"2024-03-28T02:10:12.402185+0000","last_undegraded":"2024-03-28T02:10:12.402185+0000","last_fullsized":"2024-03-28T02:10:12.402185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:22.128908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457357+0000","last_change":"2024-03-28T02:08:53.487871+0000","last_active":"2024-03-28T02:09:43.457357+0000","last_peered":"2024-03-28T02:09:43.457357+0000","last_clean":"2024-03-28T02:09:43.457357+0000","last_became_active":"2024-03-28T02:08:53.487344+0000","last_became_peered":"2024-03-28T02:08:53.487344+0000","last_unstale":"2024-03-28T02:09:43.457357+0000","last_undegraded":"2024-03-28T02:09:43.457357+0000","last_fullsized":"2024-03-28T02:09:43.457357+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:34:12.555549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:19.325967+0000","last_change":"2024-03-28T02:08:48.450507+0000","last_active":"2024-03-28T02:10:19.325967+0000","last_peered":"2024-03-28T02:10:19.325967+0000","last_clean":"2024-03-28T02:10:19.325967+0000","last_became_active":"2024-03-28T02:08:48.450065+0000","last_became_peered":"2024-03-28T02:08:48.450065+0000","last_unstale":"2024-03-28T02:10:19.325967+0000","last_undegraded":"2024-03-28T02:10:19.325967+0000","last_fullsized":"2024-03-28T02:10:19.325967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:00:45.393364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.383059+0000","last_change":"2024-03-28T02:08:49.256313+0000","last_active":"2024-03-28T02:10:12.383059+0000","last_peered":"2024-03-28T02:10:12.383059+0000","last_clean":"2024-03-28T02:10:12.383059+0000","last_became_active":"2024-03-28T02:08:49.255957+0000","last_became_peered":"2024-03-28T02:08:49.255957+0000","last_unstale":"2024-03-28T02:10:12.383059+0000","last_undegraded":"2024-03-28T02:10:12.383059+0000","last_fullsized":"2024-03-28T02:10:12.383059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:39:27.402983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457298+0000","last_change":"2024-03-28T02:08:53.498554+0000","last_active":"2024-03-28T02:09:43.457298+0000","last_peered":"2024-03-28T02:09:43.457298+0000","last_clean":"2024-03-28T02:09:43.457298+0000","last_became_active":"2024-03-28T02:08:53.498152+0000","last_became_peered":"2024-03-28T02:08:53.498152+0000","last_unstale":"2024-03-28T02:09:43.457298+0000","last_undegraded":"2024-03-28T02:09:43.457298+0000","last_fullsized":"2024-03-28T02:09:43.457298+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:41:38.234758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.463720+0000","last_change":"2024-03-28T02:08:48.450494+0000","last_active":"2024-03-28T02:09:43.463720+0000","last_peered":"2024-03-28T02:09:43.463720+0000","last_clean":"2024-03-28T02:09:43.463720+0000","last_became_active":"2024-03-28T02:08:48.450158+0000","last_became_peered":"2024-03-28T02:08:48.450158+0000","last_unstale":"2024-03-28T02:09:43.463720+0000","last_undegraded":"2024-03-28T02:09:43.463720+0000","last_fullsized":"2024-03-28T02:09:43.463720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:47:20.021116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.377248+0000","last_change":"2024-03-28T02:09:43.465511+0000","last_active":"2024-03-28T02:10:12.377248+0000","last_peered":"2024-03-28T02:10:12.377248+0000","last_clean":"2024-03-28T02:10:12.377248+0000","last_became_active":"2024-03-28T02:09:43.465096+0000","last_became_peered":"2024-03-28T02:09:43.465096+0000","last_unstale":"2024-03-28T02:10:12.377248+0000","last_undegraded":"2024-03-28T02:10:12.377248+0000","last_fullsized":"2024-03-28T02:10:12.377248+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:37.879449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460319+0000","last_change":"2024-03-28T02:08:53.490046+0000","last_active":"2024-03-28T02:09:43.460319+0000","last_peered":"2024-03-28T02:09:43.460319+0000","last_clean":"2024-03-28T02:09:43.460319+0000","last_became_active":"2024-03-28T02:08:53.489463+0000","last_became_peered":"2024-03-28T02:08:53.489463+0000","last_unstale":"2024-03-28T02:09:43.460319+0000","last_undegraded":"2024-03-28T02:09:43.460319+0000","last_fullsized":"2024-03-28T02:09:43.460319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:25:03.485568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459782+0000","last_change":"2024-03-28T02:08:48.450548+0000","last_active":"2024-03-28T02:09:43.459782+0000","last_peered":"2024-03-28T02:09:43.459782+0000","last_clean":"2024-03-28T02:09:43.459782+0000","last_became_active":"2024-03-28T02:08:48.450068+0000","last_became_peered":"2024-03-28T02:08:48.450068+0000","last_unstale":"2024-03-28T02:09:43.459782+0000","last_undegraded":"2024-03-28T02:09:43.459782+0000","last_fullsized":"2024-03-28T02:09:43.459782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:24:05.231774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.381877+0000","last_change":"2024-03-28T02:08:49.262785+0000","last_active":"2024-03-28T02:10:12.381877+0000","last_peered":"2024-03-28T02:10:12.381877+0000","last_clean":"2024-03-28T02:10:12.381877+0000","last_became_active":"2024-03-28T02:08:49.262253+0000","last_became_peered":"2024-03-28T02:08:49.262253+0000","last_unstale":"2024-03-28T02:10:12.381877+0000","last_undegraded":"2024-03-28T02:10:12.381877+0000","last_fullsized":"2024-03-28T02:10:12.381877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:07:37.064693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.464151+0000","last_change":"2024-03-28T02:08:53.483738+0000","last_active":"2024-03-28T02:09:43.464151+0000","last_peered":"2024-03-28T02:09:43.464151+0000","last_clean":"2024-03-28T02:09:43.464151+0000","last_became_active":"2024-03-28T02:08:53.483214+0000","last_became_peered":"2024-03-28T02:08:53.483214+0000","last_unstale":"2024-03-28T02:09:43.464151+0000","last_undegraded":"2024-03-28T02:09:43.464151+0000","last_fullsized":"2024-03-28T02:09:43.464151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:06:32.789839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461333+0000","last_change":"2024-03-28T02:08:48.448858+0000","last_active":"2024-03-28T02:09:43.461333+0000","last_peered":"2024-03-28T02:09:43.461333+0000","last_clean":"2024-03-28T02:09:43.461333+0000","last_became_active":"2024-03-28T02:08:48.448435+0000","last_became_peered":"2024-03-28T02:08:48.448435+0000","last_unstale":"2024-03-28T02:09:43.461333+0000","last_undegraded":"2024-03-28T02:09:43.461333+0000","last_fullsized":"2024-03-28T02:09:43.461333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:34:13.035336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.380421+0000","last_change":"2024-03-28T02:08:49.261354+0000","last_active":"2024-03-28T02:10:12.380421+0000","last_peered":"2024-03-28T02:10:12.380421+0000","last_clean":"2024-03-28T02:10:12.380421+0000","last_became_active":"2024-03-28T02:08:49.261038+0000","last_became_peered":"2024-03-28T02:08:49.261038+0000","last_unstale":"2024-03-28T02:10:12.380421+0000","last_undegraded":"2024-03-28T02:10:12.380421+0000","last_fullsized":"2024-03-28T02:10:12.380421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:03:15.615484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.464323+0000","last_change":"2024-03-28T02:08:53.488573+0000","last_active":"2024-03-28T02:09:43.464323+0000","last_peered":"2024-03-28T02:09:43.464323+0000","last_clean":"2024-03-28T02:09:43.464323+0000","last_became_active":"2024-03-28T02:08:53.488198+0000","last_became_peered":"2024-03-28T02:08:53.488198+0000","last_unstale":"2024-03-28T02:09:43.464323+0000","last_undegraded":"2024-03-28T02:09:43.464323+0000","last_fullsized":"2024-03-28T02:09:43.464323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:43:29.158223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":179,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:23.329708+0000","last_change":"2024-03-28T02:08:48.445941+0000","last_active":"2024-03-28T02:10:23.329708+0000","last_peered":"2024-03-28T02:10:23.329708+0000","last_clean":"2024-03-28T02:10:23.329708+0000","last_became_active":"2024-03-28T02:08:48.445599+0000","last_became_peered":"2024-03-28T02:08:48.445599+0000","last_unstale":"2024-03-28T02:10:23.329708+0000","last_undegraded":"2024-03-28T02:10:23.329708+0000","last_fullsized":"2024-03-28T02:10:23.329708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:28:53.524708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.401651+0000","last_change":"2024-03-28T02:08:49.258592+0000","last_active":"2024-03-28T02:10:12.401651+0000","last_peered":"2024-03-28T02:10:12.401651+0000","last_clean":"2024-03-28T02:10:12.401651+0000","last_became_active":"2024-03-28T02:08:49.256527+0000","last_became_peered":"2024-03-28T02:08:49.256527+0000","last_unstale":"2024-03-28T02:10:12.401651+0000","last_undegraded":"2024-03-28T02:10:12.401651+0000","last_fullsized":"2024-03-28T02:10:12.401651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:54:41.645210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457589+0000","last_change":"2024-03-28T02:08:53.494677+0000","last_active":"2024-03-28T02:09:43.457589+0000","last_peered":"2024-03-28T02:09:43.457589+0000","last_clean":"2024-03-28T02:09:43.457589+0000","last_became_active":"2024-03-28T02:08:53.493076+0000","last_became_peered":"2024-03-28T02:08:53.493076+0000","last_unstale":"2024-03-28T02:09:43.457589+0000","last_undegraded":"2024-03-28T02:09:43.457589+0000","last_fullsized":"2024-03-28T02:09:43.457589+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:10:44.817473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461091+0000","last_change":"2024-03-28T02:08:48.454321+0000","last_active":"2024-03-28T02:09:43.461091+0000","last_peered":"2024-03-28T02:09:43.461091+0000","last_clean":"2024-03-28T02:09:43.461091+0000","last_became_active":"2024-03-28T02:08:48.453907+0000","last_became_peered":"2024-03-28T02:08:48.453907+0000","last_unstale":"2024-03-28T02:09:43.461091+0000","last_undegraded":"2024-03-28T02:09:43.461091+0000","last_fullsized":"2024-03-28T02:09:43.461091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:45:18.347073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.395714+0000","last_change":"2024-03-28T02:08:49.262196+0000","last_active":"2024-03-28T02:10:12.395714+0000","last_peered":"2024-03-28T02:10:12.395714+0000","last_clean":"2024-03-28T02:10:12.395714+0000","last_became_active":"2024-03-28T02:08:49.261190+0000","last_became_peered":"2024-03-28T02:08:49.261190+0000","last_unstale":"2024-03-28T02:10:12.395714+0000","last_undegraded":"2024-03-28T02:10:12.395714+0000","last_fullsized":"2024-03-28T02:10:12.395714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:37:15.939535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460131+0000","last_change":"2024-03-28T02:08:53.492207+0000","last_active":"2024-03-28T02:09:43.460131+0000","last_peered":"2024-03-28T02:09:43.460131+0000","last_clean":"2024-03-28T02:09:43.460131+0000","last_became_active":"2024-03-28T02:08:53.491831+0000","last_became_peered":"2024-03-28T02:08:53.491831+0000","last_unstale":"2024-03-28T02:09:43.460131+0000","last_undegraded":"2024-03-28T02:09:43.460131+0000","last_fullsized":"2024-03-28T02:09:43.460131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:00.163397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460597+0000","last_change":"2024-03-28T02:08:48.450908+0000","last_active":"2024-03-28T02:09:43.460597+0000","last_peered":"2024-03-28T02:09:43.460597+0000","last_clean":"2024-03-28T02:09:43.460597+0000","last_became_active":"2024-03-28T02:08:48.450382+0000","last_became_peered":"2024-03-28T02:08:48.450382+0000","last_unstale":"2024-03-28T02:09:43.460597+0000","last_undegraded":"2024-03-28T02:09:43.460597+0000","last_fullsized":"2024-03-28T02:09:43.460597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:26:25.851416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.404028+0000","last_change":"2024-03-28T02:08:49.268904+0000","last_active":"2024-03-28T02:10:12.404028+0000","last_peered":"2024-03-28T02:10:12.404028+0000","last_clean":"2024-03-28T02:10:12.404028+0000","last_became_active":"2024-03-28T02:08:49.268595+0000","last_became_peered":"2024-03-28T02:08:49.268595+0000","last_unstale":"2024-03-28T02:10:12.404028+0000","last_undegraded":"2024-03-28T02:10:12.404028+0000","last_fullsized":"2024-03-28T02:10:12.404028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:12:25.903861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457690+0000","last_change":"2024-03-28T02:08:53.489115+0000","last_active":"2024-03-28T02:09:43.457690+0000","last_peered":"2024-03-28T02:09:43.457690+0000","last_clean":"2024-03-28T02:09:43.457690+0000","last_became_active":"2024-03-28T02:08:53.488716+0000","last_became_peered":"2024-03-28T02:08:53.488716+0000","last_unstale":"2024-03-28T02:09:43.457690+0000","last_undegraded":"2024-03-28T02:09:43.457690+0000","last_fullsized":"2024-03-28T02:09:43.457690+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:00:42.920205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":225,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:23.329553+0000","last_change":"2024-03-28T02:08:48.453921+0000","last_active":"2024-03-28T02:10:23.329553+0000","last_peered":"2024-03-28T02:10:23.329553+0000","last_clean":"2024-03-28T02:10:23.329553+0000","last_became_active":"2024-03-28T02:08:48.453301+0000","last_became_peered":"2024-03-28T02:08:48.453301+0000","last_unstale":"2024-03-28T02:10:23.329553+0000","last_undegraded":"2024-03-28T02:10:23.329553+0000","last_fullsized":"2024-03-28T02:10:23.329553+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:57:04.543444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66993,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":161,"num_read_kb":145,"num_write":40,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.382903+0000","last_change":"2024-03-28T02:08:49.259942+0000","last_active":"2024-03-28T02:10:12.382903+0000","last_peered":"2024-03-28T02:10:12.382903+0000","last_clean":"2024-03-28T02:10:12.382903+0000","last_became_active":"2024-03-28T02:08:49.259602+0000","last_became_peered":"2024-03-28T02:08:49.259602+0000","last_unstale":"2024-03-28T02:10:12.382903+0000","last_undegraded":"2024-03-28T02:10:12.382903+0000","last_fullsized":"2024-03-28T02:10:12.382903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:38:05.897720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461108+0000","last_change":"2024-03-28T02:08:53.495675+0000","last_active":"2024-03-28T02:09:43.461108+0000","last_peered":"2024-03-28T02:09:43.461108+0000","last_clean":"2024-03-28T02:09:43.461108+0000","last_became_active":"2024-03-28T02:08:53.495233+0000","last_became_peered":"2024-03-28T02:08:53.495233+0000","last_unstale":"2024-03-28T02:09:43.461108+0000","last_undegraded":"2024-03-28T02:09:43.461108+0000","last_fullsized":"2024-03-28T02:09:43.461108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:43.805151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.488362+0000","last_change":"2024-03-28T02:09:43.488079+0000","last_active":"2024-03-28T02:09:43.488362+0000","last_peered":"2024-03-28T02:09:43.488362+0000","last_clean":"2024-03-28T02:09:43.488362+0000","last_became_active":"2024-03-28T02:09:43.467010+0000","last_became_peered":"2024-03-28T02:09:43.467010+0000","last_unstale":"2024-03-28T02:09:43.488362+0000","last_undegraded":"2024-03-28T02:09:43.488362+0000","last_fullsized":"2024-03-28T02:09:43.488362+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:47:56.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388205+0000","last_change":"2024-03-28T02:08:49.259179+0000","last_active":"2024-03-28T02:10:12.388205+0000","last_peered":"2024-03-28T02:10:12.388205+0000","last_clean":"2024-03-28T02:10:12.388205+0000","last_became_active":"2024-03-28T02:08:49.258538+0000","last_became_peered":"2024-03-28T02:08:49.258538+0000","last_unstale":"2024-03-28T02:10:12.388205+0000","last_undegraded":"2024-03-28T02:10:12.388205+0000","last_fullsized":"2024-03-28T02:10:12.388205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:12:20.008528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.458972+0000","last_change":"2024-03-28T02:08:53.499523+0000","last_active":"2024-03-28T02:09:43.458972+0000","last_peered":"2024-03-28T02:09:43.458972+0000","last_clean":"2024-03-28T02:09:43.458972+0000","last_became_active":"2024-03-28T02:08:53.499112+0000","last_became_peered":"2024-03-28T02:08:53.499112+0000","last_unstale":"2024-03-28T02:09:43.458972+0000","last_undegraded":"2024-03-28T02:09:43.458972+0000","last_fullsized":"2024-03-28T02:09:43.458972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:31:12.008640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459676+0000","last_change":"2024-03-28T02:08:48.452952+0000","last_active":"2024-03-28T02:09:43.459676+0000","last_peered":"2024-03-28T02:09:43.459676+0000","last_clean":"2024-03-28T02:09:43.459676+0000","last_became_active":"2024-03-28T02:08:48.452586+0000","last_became_peered":"2024-03-28T02:08:48.452586+0000","last_unstale":"2024-03-28T02:09:43.459676+0000","last_undegraded":"2024-03-28T02:09:43.459676+0000","last_fullsized":"2024-03-28T02:09:43.459676+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:44.749902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.391810+0000","last_change":"2024-03-28T02:08:49.256746+0000","last_active":"2024-03-28T02:10:12.391810+0000","last_peered":"2024-03-28T02:10:12.391810+0000","last_clean":"2024-03-28T02:10:12.391810+0000","last_became_active":"2024-03-28T02:08:49.256398+0000","last_became_peered":"2024-03-28T02:08:49.256398+0000","last_unstale":"2024-03-28T02:10:12.391810+0000","last_undegraded":"2024-03-28T02:10:12.391810+0000","last_fullsized":"2024-03-28T02:10:12.391810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:10:52.021218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.458352+0000","last_change":"2024-03-28T02:08:53.494257+0000","last_active":"2024-03-28T02:09:43.458352+0000","last_peered":"2024-03-28T02:09:43.458352+0000","last_clean":"2024-03-28T02:09:43.458352+0000","last_became_active":"2024-03-28T02:08:53.493889+0000","last_became_peered":"2024-03-28T02:08:53.493889+0000","last_unstale":"2024-03-28T02:09:43.458352+0000","last_undegraded":"2024-03-28T02:09:43.458352+0000","last_fullsized":"2024-03-28T02:09:43.458352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:49.292686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:19.376762+0000","last_change":"2024-03-28T02:08:48.447361+0000","last_active":"2024-03-28T02:10:19.376762+0000","last_peered":"2024-03-28T02:10:19.376762+0000","last_clean":"2024-03-28T02:10:19.376762+0000","last_became_active":"2024-03-28T02:08:48.446943+0000","last_became_peered":"2024-03-28T02:08:48.446943+0000","last_unstale":"2024-03-28T02:10:19.376762+0000","last_undegraded":"2024-03-28T02:10:19.376762+0000","last_fullsized":"2024-03-28T02:10:19.376762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:19:46.552447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388883+0000","last_change":"2024-03-28T02:08:49.255253+0000","last_active":"2024-03-28T02:10:12.388883+0000","last_peered":"2024-03-28T02:10:12.388883+0000","last_clean":"2024-03-28T02:10:12.388883+0000","last_became_active":"2024-03-28T02:08:49.254804+0000","last_became_peered":"2024-03-28T02:08:49.254804+0000","last_unstale":"2024-03-28T02:10:12.388883+0000","last_undegraded":"2024-03-28T02:10:12.388883+0000","last_fullsized":"2024-03-28T02:10:12.388883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:53:31.927830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.466912+0000","last_change":"2024-03-28T02:09:43.466912+0000","last_active":"2024-03-28T02:09:43.466912+0000","last_peered":"2024-03-28T02:09:43.466912+0000","last_clean":"2024-03-28T02:09:43.466912+0000","last_became_active":"2024-03-28T02:09:43.466462+0000","last_became_peered":"2024-03-28T02:09:43.466462+0000","last_unstale":"2024-03-28T02:09:43.466912+0000","last_undegraded":"2024-03-28T02:09:43.466912+0000","last_fullsized":"2024-03-28T02:09:43.466912+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:51:20.559601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.480457+0000","last_change":"2024-03-28T02:09:43.480082+0000","last_active":"2024-03-28T02:09:43.480457+0000","last_peered":"2024-03-28T02:09:43.480457+0000","last_clean":"2024-03-28T02:09:43.480457+0000","last_became_active":"2024-03-28T02:09:43.464495+0000","last_became_peered":"2024-03-28T02:09:43.464495+0000","last_unstale":"2024-03-28T02:09:43.480457+0000","last_undegraded":"2024-03-28T02:09:43.480457+0000","last_fullsized":"2024-03-28T02:09:43.480457+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:55:11.451630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.401459+0000","last_change":"2024-03-28T02:09:43.464753+0000","last_active":"2024-03-28T02:10:12.401459+0000","last_peered":"2024-03-28T02:10:12.401459+0000","last_clean":"2024-03-28T02:10:12.401459+0000","last_became_active":"2024-03-28T02:09:43.464400+0000","last_became_peered":"2024-03-28T02:09:43.464400+0000","last_unstale":"2024-03-28T02:10:12.401459+0000","last_undegraded":"2024-03-28T02:10:12.401459+0000","last_fullsized":"2024-03-28T02:10:12.401459+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:19:35.611109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460947+0000","last_change":"2024-03-28T02:08:53.490855+0000","last_active":"2024-03-28T02:09:43.460947+0000","last_peered":"2024-03-28T02:09:43.460947+0000","last_clean":"2024-03-28T02:09:43.460947+0000","last_became_active":"2024-03-28T02:08:53.490415+0000","last_became_peered":"2024-03-28T02:08:53.490415+0000","last_unstale":"2024-03-28T02:09:43.460947+0000","last_undegraded":"2024-03-28T02:09:43.460947+0000","last_fullsized":"2024-03-28T02:09:43.460947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:56:37.426398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.463168+0000","last_change":"2024-03-28T02:08:48.448909+0000","last_active":"2024-03-28T02:09:43.463168+0000","last_peered":"2024-03-28T02:09:43.463168+0000","last_clean":"2024-03-28T02:09:43.463168+0000","last_became_active":"2024-03-28T02:08:48.448548+0000","last_became_peered":"2024-03-28T02:08:48.448548+0000","last_unstale":"2024-03-28T02:09:43.463168+0000","last_undegraded":"2024-03-28T02:09:43.463168+0000","last_fullsized":"2024-03-28T02:09:43.463168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:51:06.206266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390639+0000","last_change":"2024-03-28T02:08:49.266920+0000","last_active":"2024-03-28T02:10:12.390639+0000","last_peered":"2024-03-28T02:10:12.390639+0000","last_clean":"2024-03-28T02:10:12.390639+0000","last_became_active":"2024-03-28T02:08:49.266600+0000","last_became_peered":"2024-03-28T02:08:49.266600+0000","last_unstale":"2024-03-28T02:10:12.390639+0000","last_undegraded":"2024-03-28T02:10:12.390639+0000","last_fullsized":"2024-03-28T02:10:12.390639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:57.735460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459737+0000","last_change":"2024-03-28T02:08:53.631636+0000","last_active":"2024-03-28T02:09:43.459737+0000","last_peered":"2024-03-28T02:09:43.459737+0000","last_clean":"2024-03-28T02:09:43.459737+0000","last_became_active":"2024-03-28T02:08:53.631183+0000","last_became_peered":"2024-03-28T02:08:53.631183+0000","last_unstale":"2024-03-28T02:09:43.459737+0000","last_undegraded":"2024-03-28T02:09:43.459737+0000","last_fullsized":"2024-03-28T02:09:43.459737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:22:09.212726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.458121+0000","last_change":"2024-03-28T02:08:48.440789+0000","last_active":"2024-03-28T02:09:43.458121+0000","last_peered":"2024-03-28T02:09:43.458121+0000","last_clean":"2024-03-28T02:09:43.458121+0000","last_became_active":"2024-03-28T02:08:48.440403+0000","last_became_peered":"2024-03-28T02:08:48.440403+0000","last_unstale":"2024-03-28T02:09:43.458121+0000","last_undegraded":"2024-03-28T02:09:43.458121+0000","last_fullsized":"2024-03-28T02:09:43.458121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:37.227279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.378816+0000","last_change":"2024-03-28T02:08:49.255340+0000","last_active":"2024-03-28T02:10:12.378816+0000","last_peered":"2024-03-28T02:10:12.378816+0000","last_clean":"2024-03-28T02:10:12.378816+0000","last_became_active":"2024-03-28T02:08:49.255027+0000","last_became_peered":"2024-03-28T02:08:49.255027+0000","last_unstale":"2024-03-28T02:10:12.378816+0000","last_undegraded":"2024-03-28T02:10:12.378816+0000","last_fullsized":"2024-03-28T02:10:12.378816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:54:03.454022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460139+0000","last_change":"2024-03-28T02:08:53.499721+0000","last_active":"2024-03-28T02:09:43.460139+0000","last_peered":"2024-03-28T02:09:43.460139+0000","last_clean":"2024-03-28T02:09:43.460139+0000","last_became_active":"2024-03-28T02:08:53.499117+0000","last_became_peered":"2024-03-28T02:08:53.499117+0000","last_unstale":"2024-03-28T02:09:43.460139+0000","last_undegraded":"2024-03-28T02:09:43.460139+0000","last_fullsized":"2024-03-28T02:09:43.460139+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:07:33.561249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:19.377568+0000","last_change":"2024-03-28T02:08:48.453592+0000","last_active":"2024-03-28T02:10:19.377568+0000","last_peered":"2024-03-28T02:10:19.377568+0000","last_clean":"2024-03-28T02:10:19.377568+0000","last_became_active":"2024-03-28T02:08:48.453169+0000","last_became_peered":"2024-03-28T02:08:48.453169+0000","last_unstale":"2024-03-28T02:10:19.377568+0000","last_undegraded":"2024-03-28T02:10:19.377568+0000","last_fullsized":"2024-03-28T02:10:19.377568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:33.163170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390591+0000","last_change":"2024-03-28T02:08:49.263143+0000","last_active":"2024-03-28T02:10:12.390591+0000","last_peered":"2024-03-28T02:10:12.390591+0000","last_clean":"2024-03-28T02:10:12.390591+0000","last_became_active":"2024-03-28T02:08:49.262829+0000","last_became_peered":"2024-03-28T02:08:49.262829+0000","last_unstale":"2024-03-28T02:10:12.390591+0000","last_undegraded":"2024-03-28T02:10:12.390591+0000","last_fullsized":"2024-03-28T02:10:12.390591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:01:07.809685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460229+0000","last_change":"2024-03-28T02:08:53.631595+0000","last_active":"2024-03-28T02:09:43.460229+0000","last_peered":"2024-03-28T02:09:43.460229+0000","last_clean":"2024-03-28T02:09:43.460229+0000","last_became_active":"2024-03-28T02:08:53.630863+0000","last_became_peered":"2024-03-28T02:08:53.630863+0000","last_unstale":"2024-03-28T02:09:43.460229+0000","last_undegraded":"2024-03-28T02:09:43.460229+0000","last_fullsized":"2024-03-28T02:09:43.460229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:38:44.712728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390396+0000","last_change":"2024-03-28T02:08:49.263523+0000","last_active":"2024-03-28T02:10:12.390396+0000","last_peered":"2024-03-28T02:10:12.390396+0000","last_clean":"2024-03-28T02:10:12.390396+0000","last_became_active":"2024-03-28T02:08:49.263079+0000","last_became_peered":"2024-03-28T02:08:49.263079+0000","last_unstale":"2024-03-28T02:10:12.390396+0000","last_undegraded":"2024-03-28T02:10:12.390396+0000","last_fullsized":"2024-03-28T02:10:12.390396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:15:37.516032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:19.376270+0000","last_change":"2024-03-28T02:08:48.455723+0000","last_active":"2024-03-28T02:10:19.376270+0000","last_peered":"2024-03-28T02:10:19.376270+0000","last_clean":"2024-03-28T02:10:19.376270+0000","last_became_active":"2024-03-28T02:08:48.455340+0000","last_became_peered":"2024-03-28T02:08:48.455340+0000","last_unstale":"2024-03-28T02:10:19.376270+0000","last_undegraded":"2024-03-28T02:10:19.376270+0000","last_fullsized":"2024-03-28T02:10:19.376270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:15:36.376221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.470892+0000","last_change":"2024-03-28T02:09:43.470892+0000","last_active":"2024-03-28T02:09:43.470892+0000","last_peered":"2024-03-28T02:09:43.470892+0000","last_clean":"2024-03-28T02:09:43.470892+0000","last_became_active":"2024-03-28T02:09:43.465112+0000","last_became_peered":"2024-03-28T02:09:43.465112+0000","last_unstale":"2024-03-28T02:09:43.470892+0000","last_undegraded":"2024-03-28T02:09:43.470892+0000","last_fullsized":"2024-03-28T02:09:43.470892+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:59:02.285845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.384529+0000","last_change":"2024-03-28T02:08:49.260380+0000","last_active":"2024-03-28T02:10:12.384529+0000","last_peered":"2024-03-28T02:10:12.384529+0000","last_clean":"2024-03-28T02:10:12.384529+0000","last_became_active":"2024-03-28T02:08:49.260012+0000","last_became_peered":"2024-03-28T02:08:49.260012+0000","last_unstale":"2024-03-28T02:10:12.384529+0000","last_undegraded":"2024-03-28T02:10:12.384529+0000","last_fullsized":"2024-03-28T02:10:12.384529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:46:31.233692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459991+0000","last_change":"2024-03-28T02:08:48.447659+0000","last_active":"2024-03-28T02:09:43.459991+0000","last_peered":"2024-03-28T02:09:43.459991+0000","last_clean":"2024-03-28T02:09:43.459991+0000","last_became_active":"2024-03-28T02:08:48.447263+0000","last_became_peered":"2024-03-28T02:08:48.447263+0000","last_unstale":"2024-03-28T02:09:43.459991+0000","last_undegraded":"2024-03-28T02:09:43.459991+0000","last_fullsized":"2024-03-28T02:09:43.459991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:16:32.453528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460498+0000","last_change":"2024-03-28T02:07:33.021769+0000","last_active":"2024-03-28T02:09:43.460498+0000","last_peered":"2024-03-28T02:09:43.460498+0000","last_clean":"2024-03-28T02:09:43.460498+0000","last_became_active":"2024-03-28T02:07:33.021338+0000","last_became_peered":"2024-03-28T02:07:33.021338+0000","last_unstale":"2024-03-28T02:09:43.460498+0000","last_undegraded":"2024-03-28T02:09:43.460498+0000","last_fullsized":"2024-03-28T02:09:43.460498+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:04:07.152098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:04:07.152098+0000","last_clean_scrub_stamp":"2024-03-28T02:04:07.152098+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:01:39.981299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461337+0000","last_change":"2024-03-28T02:08:53.631687+0000","last_active":"2024-03-28T02:09:43.461337+0000","last_peered":"2024-03-28T02:09:43.461337+0000","last_clean":"2024-03-28T02:09:43.461337+0000","last_became_active":"2024-03-28T02:08:53.631291+0000","last_became_peered":"2024-03-28T02:08:53.631291+0000","last_unstale":"2024-03-28T02:09:43.461337+0000","last_undegraded":"2024-03-28T02:09:43.461337+0000","last_fullsized":"2024-03-28T02:09:43.461337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:18:51.563667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.387552+0000","last_change":"2024-03-28T02:08:49.255310+0000","last_active":"2024-03-28T02:10:12.387552+0000","last_peered":"2024-03-28T02:10:12.387552+0000","last_clean":"2024-03-28T02:10:12.387552+0000","last_became_active":"2024-03-28T02:08:49.254923+0000","last_became_peered":"2024-03-28T02:08:49.254923+0000","last_unstale":"2024-03-28T02:10:12.387552+0000","last_undegraded":"2024-03-28T02:10:12.387552+0000","last_fullsized":"2024-03-28T02:10:12.387552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:34.660601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460521+0000","last_change":"2024-03-28T02:08:48.457646+0000","last_active":"2024-03-28T02:09:43.460521+0000","last_peered":"2024-03-28T02:09:43.460521+0000","last_clean":"2024-03-28T02:09:43.460521+0000","last_became_active":"2024-03-28T02:08:48.457344+0000","last_became_peered":"2024-03-28T02:08:48.457344+0000","last_unstale":"2024-03-28T02:09:43.460521+0000","last_undegraded":"2024-03-28T02:09:43.460521+0000","last_fullsized":"2024-03-28T02:09:43.460521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:25:19.077816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460105+0000","last_change":"2024-03-28T02:08:53.486320+0000","last_active":"2024-03-28T02:09:43.460105+0000","last_peered":"2024-03-28T02:09:43.460105+0000","last_clean":"2024-03-28T02:09:43.460105+0000","last_became_active":"2024-03-28T02:08:53.485771+0000","last_became_peered":"2024-03-28T02:08:53.485771+0000","last_unstale":"2024-03-28T02:09:43.460105+0000","last_undegraded":"2024-03-28T02:09:43.460105+0000","last_fullsized":"2024-03-28T02:09:43.460105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:48.452990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461719+0000","last_change":"2024-03-28T02:08:48.458854+0000","last_active":"2024-03-28T02:09:43.461719+0000","last_peered":"2024-03-28T02:09:43.461719+0000","last_clean":"2024-03-28T02:09:43.461719+0000","last_became_active":"2024-03-28T02:08:48.456667+0000","last_became_peered":"2024-03-28T02:08:48.456667+0000","last_unstale":"2024-03-28T02:09:43.461719+0000","last_undegraded":"2024-03-28T02:09:43.461719+0000","last_fullsized":"2024-03-28T02:09:43.461719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:37:54.113797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.391088+0000","last_change":"2024-03-28T02:08:49.255300+0000","last_active":"2024-03-28T02:10:12.391088+0000","last_peered":"2024-03-28T02:10:12.391088+0000","last_clean":"2024-03-28T02:10:12.391088+0000","last_became_active":"2024-03-28T02:08:49.254955+0000","last_became_peered":"2024-03-28T02:08:49.254955+0000","last_unstale":"2024-03-28T02:10:12.391088+0000","last_undegraded":"2024-03-28T02:10:12.391088+0000","last_fullsized":"2024-03-28T02:10:12.391088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:29.093770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.465003+0000","last_change":"2024-03-28T02:08:53.631966+0000","last_active":"2024-03-28T02:09:43.465003+0000","last_peered":"2024-03-28T02:09:43.465003+0000","last_clean":"2024-03-28T02:09:43.465003+0000","last_became_active":"2024-03-28T02:08:53.631457+0000","last_became_peered":"2024-03-28T02:08:53.631457+0000","last_unstale":"2024-03-28T02:09:43.465003+0000","last_undegraded":"2024-03-28T02:09:43.465003+0000","last_fullsized":"2024-03-28T02:09:43.465003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:36:46.623567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.458395+0000","last_change":"2024-03-28T02:08:48.452991+0000","last_active":"2024-03-28T02:09:43.458395+0000","last_peered":"2024-03-28T02:09:43.458395+0000","last_clean":"2024-03-28T02:09:43.458395+0000","last_became_active":"2024-03-28T02:08:48.452248+0000","last_became_peered":"2024-03-28T02:08:48.452248+0000","last_unstale":"2024-03-28T02:09:43.458395+0000","last_undegraded":"2024-03-28T02:09:43.458395+0000","last_fullsized":"2024-03-28T02:09:43.458395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:13:40.100006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.389316+0000","last_change":"2024-03-28T02:08:49.259575+0000","last_active":"2024-03-28T02:10:12.389316+0000","last_peered":"2024-03-28T02:10:12.389316+0000","last_clean":"2024-03-28T02:10:12.389316+0000","last_became_active":"2024-03-28T02:08:49.259255+0000","last_became_peered":"2024-03-28T02:08:49.259255+0000","last_unstale":"2024-03-28T02:10:12.389316+0000","last_undegraded":"2024-03-28T02:10:12.389316+0000","last_fullsized":"2024-03-28T02:10:12.389316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:47:17.741691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457433+0000","last_change":"2024-03-28T02:08:53.488304+0000","last_active":"2024-03-28T02:09:43.457433+0000","last_peered":"2024-03-28T02:09:43.457433+0000","last_clean":"2024-03-28T02:09:43.457433+0000","last_became_active":"2024-03-28T02:08:53.487783+0000","last_became_peered":"2024-03-28T02:08:53.487783+0000","last_unstale":"2024-03-28T02:09:43.457433+0000","last_undegraded":"2024-03-28T02:09:43.457433+0000","last_fullsized":"2024-03-28T02:09:43.457433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:54:50.494685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460346+0000","last_change":"2024-03-28T02:08:48.450795+0000","last_active":"2024-03-28T02:09:43.460346+0000","last_peered":"2024-03-28T02:09:43.460346+0000","last_clean":"2024-03-28T02:09:43.460346+0000","last_became_active":"2024-03-28T02:08:48.450449+0000","last_became_peered":"2024-03-28T02:08:48.450449+0000","last_unstale":"2024-03-28T02:09:43.460346+0000","last_undegraded":"2024-03-28T02:09:43.460346+0000","last_fullsized":"2024-03-28T02:09:43.460346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:44.576188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.393462+0000","last_change":"2024-03-28T02:08:49.264220+0000","last_active":"2024-03-28T02:10:12.393462+0000","last_peered":"2024-03-28T02:10:12.393462+0000","last_clean":"2024-03-28T02:10:12.393462+0000","last_became_active":"2024-03-28T02:08:49.263757+0000","last_became_peered":"2024-03-28T02:08:49.263757+0000","last_unstale":"2024-03-28T02:10:12.393462+0000","last_undegraded":"2024-03-28T02:10:12.393462+0000","last_fullsized":"2024-03-28T02:10:12.393462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:20:44.000256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459943+0000","last_change":"2024-03-28T02:08:53.631904+0000","last_active":"2024-03-28T02:09:43.459943+0000","last_peered":"2024-03-28T02:09:43.459943+0000","last_clean":"2024-03-28T02:09:43.459943+0000","last_became_active":"2024-03-28T02:08:53.631263+0000","last_became_peered":"2024-03-28T02:08:53.631263+0000","last_unstale":"2024-03-28T02:09:43.459943+0000","last_undegraded":"2024-03-28T02:09:43.459943+0000","last_fullsized":"2024-03-28T02:09:43.459943+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:30:53.511632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457483+0000","last_change":"2024-03-28T02:08:48.450272+0000","last_active":"2024-03-28T02:09:43.457483+0000","last_peered":"2024-03-28T02:09:43.457483+0000","last_clean":"2024-03-28T02:09:43.457483+0000","last_became_active":"2024-03-28T02:08:48.449905+0000","last_became_peered":"2024-03-28T02:08:48.449905+0000","last_unstale":"2024-03-28T02:09:43.457483+0000","last_undegraded":"2024-03-28T02:09:43.457483+0000","last_fullsized":"2024-03-28T02:09:43.457483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:56:49.493569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.353622+0000","last_change":"2024-03-28T02:08:49.256816+0000","last_active":"2024-03-28T02:10:12.353622+0000","last_peered":"2024-03-28T02:10:12.353622+0000","last_clean":"2024-03-28T02:10:12.353622+0000","last_became_active":"2024-03-28T02:08:49.256369+0000","last_became_peered":"2024-03-28T02:08:49.256369+0000","last_unstale":"2024-03-28T02:10:12.353622+0000","last_undegraded":"2024-03-28T02:10:12.353622+0000","last_fullsized":"2024-03-28T02:10:12.353622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:39.714619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.465990+0000","last_change":"2024-03-28T02:09:43.465595+0000","last_active":"2024-03-28T02:09:43.465990+0000","last_peered":"2024-03-28T02:09:43.465990+0000","last_clean":"2024-03-28T02:09:43.465990+0000","last_became_active":"2024-03-28T02:09:43.465177+0000","last_became_peered":"2024-03-28T02:09:43.465177+0000","last_unstale":"2024-03-28T02:09:43.465990+0000","last_undegraded":"2024-03-28T02:09:43.465990+0000","last_fullsized":"2024-03-28T02:09:43.465990+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:25:07.219000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.464143+0000","last_change":"2024-03-28T02:08:48.454998+0000","last_active":"2024-03-28T02:09:43.464143+0000","last_peered":"2024-03-28T02:09:43.464143+0000","last_clean":"2024-03-28T02:09:43.464143+0000","last_became_active":"2024-03-28T02:08:48.454674+0000","last_became_peered":"2024-03-28T02:08:48.454674+0000","last_unstale":"2024-03-28T02:09:43.464143+0000","last_undegraded":"2024-03-28T02:09:43.464143+0000","last_fullsized":"2024-03-28T02:09:43.464143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:01:23.432625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.384467+0000","last_change":"2024-03-28T02:08:49.263714+0000","last_active":"2024-03-28T02:10:12.384467+0000","last_peered":"2024-03-28T02:10:12.384467+0000","last_clean":"2024-03-28T02:10:12.384467+0000","last_became_active":"2024-03-28T02:08:49.263335+0000","last_became_peered":"2024-03-28T02:08:49.263335+0000","last_unstale":"2024-03-28T02:10:12.384467+0000","last_undegraded":"2024-03-28T02:10:12.384467+0000","last_fullsized":"2024-03-28T02:10:12.384467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:28:13.153319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459831+0000","last_change":"2024-03-28T02:08:53.496751+0000","last_active":"2024-03-28T02:09:43.459831+0000","last_peered":"2024-03-28T02:09:43.459831+0000","last_clean":"2024-03-28T02:09:43.459831+0000","last_became_active":"2024-03-28T02:08:53.496393+0000","last_became_peered":"2024-03-28T02:08:53.496393+0000","last_unstale":"2024-03-28T02:09:43.459831+0000","last_undegraded":"2024-03-28T02:09:43.459831+0000","last_fullsized":"2024-03-28T02:09:43.459831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:44:13.268278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457433+0000","last_change":"2024-03-28T02:08:48.456026+0000","last_active":"2024-03-28T02:09:43.457433+0000","last_peered":"2024-03-28T02:09:43.457433+0000","last_clean":"2024-03-28T02:09:43.457433+0000","last_became_active":"2024-03-28T02:08:48.455264+0000","last_became_peered":"2024-03-28T02:08:48.455264+0000","last_unstale":"2024-03-28T02:09:43.457433+0000","last_undegraded":"2024-03-28T02:09:43.457433+0000","last_fullsized":"2024-03-28T02:09:43.457433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:21:51.682413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.386231+0000","last_change":"2024-03-28T02:08:49.260416+0000","last_active":"2024-03-28T02:10:12.386231+0000","last_peered":"2024-03-28T02:10:12.386231+0000","last_clean":"2024-03-28T02:10:12.386231+0000","last_became_active":"2024-03-28T02:08:49.260071+0000","last_became_peered":"2024-03-28T02:08:49.260071+0000","last_unstale":"2024-03-28T02:10:12.386231+0000","last_undegraded":"2024-03-28T02:10:12.386231+0000","last_fullsized":"2024-03-28T02:10:12.386231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:34:56.414305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459713+0000","last_change":"2024-03-28T02:08:53.631633+0000","last_active":"2024-03-28T02:09:43.459713+0000","last_peered":"2024-03-28T02:09:43.459713+0000","last_clean":"2024-03-28T02:09:43.459713+0000","last_became_active":"2024-03-28T02:08:53.631158+0000","last_became_peered":"2024-03-28T02:08:53.631158+0000","last_unstale":"2024-03-28T02:09:43.459713+0000","last_undegraded":"2024-03-28T02:09:43.459713+0000","last_fullsized":"2024-03-28T02:09:43.459713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:37:43.749860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.471538+0000","last_change":"2024-03-28T02:09:43.471538+0000","last_active":"2024-03-28T02:09:43.471538+0000","last_peered":"2024-03-28T02:09:43.471538+0000","last_clean":"2024-03-28T02:09:43.471538+0000","last_became_active":"2024-03-28T02:09:43.466626+0000","last_became_peered":"2024-03-28T02:09:43.466626+0000","last_unstale":"2024-03-28T02:09:43.471538+0000","last_undegraded":"2024-03-28T02:09:43.471538+0000","last_fullsized":"2024-03-28T02:09:43.471538+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:07:15.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402156+0000","last_change":"2024-03-28T02:08:49.268529+0000","last_active":"2024-03-28T02:10:12.402156+0000","last_peered":"2024-03-28T02:10:12.402156+0000","last_clean":"2024-03-28T02:10:12.402156+0000","last_became_active":"2024-03-28T02:08:49.267869+0000","last_became_peered":"2024-03-28T02:08:49.267869+0000","last_unstale":"2024-03-28T02:10:12.402156+0000","last_undegraded":"2024-03-28T02:10:12.402156+0000","last_fullsized":"2024-03-28T02:10:12.402156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:52:34.633616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457531+0000","last_change":"2024-03-28T02:08:53.495894+0000","last_active":"2024-03-28T02:09:43.457531+0000","last_peered":"2024-03-28T02:09:43.457531+0000","last_clean":"2024-03-28T02:09:43.457531+0000","last_became_active":"2024-03-28T02:08:53.495442+0000","last_became_peered":"2024-03-28T02:08:53.495442+0000","last_unstale":"2024-03-28T02:09:43.457531+0000","last_undegraded":"2024-03-28T02:09:43.457531+0000","last_fullsized":"2024-03-28T02:09:43.457531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:32:33.318061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:19.383470+0000","last_change":"2024-03-28T02:08:48.450376+0000","last_active":"2024-03-28T02:10:19.383470+0000","last_peered":"2024-03-28T02:10:19.383470+0000","last_clean":"2024-03-28T02:10:19.383470+0000","last_became_active":"2024-03-28T02:08:48.449978+0000","last_became_peered":"2024-03-28T02:08:48.449978+0000","last_unstale":"2024-03-28T02:10:19.383470+0000","last_undegraded":"2024-03-28T02:10:19.383470+0000","last_fullsized":"2024-03-28T02:10:19.383470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:33:56.284025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402305+0000","last_change":"2024-03-28T02:08:49.269011+0000","last_active":"2024-03-28T02:10:12.402305+0000","last_peered":"2024-03-28T02:10:12.402305+0000","last_clean":"2024-03-28T02:10:12.402305+0000","last_became_active":"2024-03-28T02:08:49.268141+0000","last_became_peered":"2024-03-28T02:08:49.268141+0000","last_unstale":"2024-03-28T02:10:12.402305+0000","last_undegraded":"2024-03-28T02:10:12.402305+0000","last_fullsized":"2024-03-28T02:10:12.402305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:54:32.708445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.463419+0000","last_change":"2024-03-28T02:08:53.501549+0000","last_active":"2024-03-28T02:09:43.463419+0000","last_peered":"2024-03-28T02:09:43.463419+0000","last_clean":"2024-03-28T02:09:43.463419+0000","last_became_active":"2024-03-28T02:08:53.501017+0000","last_became_peered":"2024-03-28T02:08:53.501017+0000","last_unstale":"2024-03-28T02:09:43.463419+0000","last_undegraded":"2024-03-28T02:09:43.463419+0000","last_fullsized":"2024-03-28T02:09:43.463419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:33:50.346117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460819+0000","last_change":"2024-03-28T02:08:48.454671+0000","last_active":"2024-03-28T02:09:43.460819+0000","last_peered":"2024-03-28T02:09:43.460819+0000","last_clean":"2024-03-28T02:09:43.460819+0000","last_became_active":"2024-03-28T02:08:48.454329+0000","last_became_peered":"2024-03-28T02:08:48.454329+0000","last_unstale":"2024-03-28T02:09:43.460819+0000","last_undegraded":"2024-03-28T02:09:43.460819+0000","last_fullsized":"2024-03-28T02:09:43.460819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:41:24.087559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388318+0000","last_change":"2024-03-28T02:08:49.259510+0000","last_active":"2024-03-28T02:10:12.388318+0000","last_peered":"2024-03-28T02:10:12.388318+0000","last_clean":"2024-03-28T02:10:12.388318+0000","last_became_active":"2024-03-28T02:08:49.259141+0000","last_became_peered":"2024-03-28T02:08:49.259141+0000","last_unstale":"2024-03-28T02:10:12.388318+0000","last_undegraded":"2024-03-28T02:10:12.388318+0000","last_fullsized":"2024-03-28T02:10:12.388318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:56:50.754576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461174+0000","last_change":"2024-03-28T02:08:53.488792+0000","last_active":"2024-03-28T02:09:43.461174+0000","last_peered":"2024-03-28T02:09:43.461174+0000","last_clean":"2024-03-28T02:09:43.461174+0000","last_became_active":"2024-03-28T02:08:53.488013+0000","last_became_peered":"2024-03-28T02:08:53.488013+0000","last_unstale":"2024-03-28T02:09:43.461174+0000","last_undegraded":"2024-03-28T02:09:43.461174+0000","last_fullsized":"2024-03-28T02:09:43.461174+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:39:52.006618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460300+0000","last_change":"2024-03-28T02:08:48.454609+0000","last_active":"2024-03-28T02:09:43.460300+0000","last_peered":"2024-03-28T02:09:43.460300+0000","last_clean":"2024-03-28T02:09:43.460300+0000","last_became_active":"2024-03-28T02:08:48.454297+0000","last_became_peered":"2024-03-28T02:08:48.454297+0000","last_unstale":"2024-03-28T02:09:43.460300+0000","last_undegraded":"2024-03-28T02:09:43.460300+0000","last_fullsized":"2024-03-28T02:09:43.460300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:31:42.819338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.379569+0000","last_change":"2024-03-28T02:08:49.263700+0000","last_active":"2024-03-28T02:10:12.379569+0000","last_peered":"2024-03-28T02:10:12.379569+0000","last_clean":"2024-03-28T02:10:12.379569+0000","last_became_active":"2024-03-28T02:08:49.263073+0000","last_became_peered":"2024-03-28T02:08:49.263073+0000","last_unstale":"2024-03-28T02:10:12.379569+0000","last_undegraded":"2024-03-28T02:10:12.379569+0000","last_fullsized":"2024-03-28T02:10:12.379569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:34:19.158284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.389095+0000","last_change":"2024-03-28T02:08:49.265572+0000","last_active":"2024-03-28T02:10:12.389095+0000","last_peered":"2024-03-28T02:10:12.389095+0000","last_clean":"2024-03-28T02:10:12.389095+0000","last_became_active":"2024-03-28T02:08:49.265202+0000","last_became_peered":"2024-03-28T02:08:49.265202+0000","last_unstale":"2024-03-28T02:10:12.389095+0000","last_undegraded":"2024-03-28T02:10:12.389095+0000","last_fullsized":"2024-03-28T02:10:12.389095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:00:05.296762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.396692+0000","last_change":"2024-03-28T02:08:49.266529+0000","last_active":"2024-03-28T02:10:12.396692+0000","last_peered":"2024-03-28T02:10:12.396692+0000","last_clean":"2024-03-28T02:10:12.396692+0000","last_became_active":"2024-03-28T02:08:49.266162+0000","last_became_peered":"2024-03-28T02:08:49.266162+0000","last_unstale":"2024-03-28T02:10:12.396692+0000","last_undegraded":"2024-03-28T02:10:12.396692+0000","last_fullsized":"2024-03-28T02:10:12.396692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:31:40.117845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.392576+0000","last_change":"2024-03-28T02:08:49.264712+0000","last_active":"2024-03-28T02:10:12.392576+0000","last_peered":"2024-03-28T02:10:12.392576+0000","last_clean":"2024-03-28T02:10:12.392576+0000","last_became_active":"2024-03-28T02:08:49.264363+0000","last_became_peered":"2024-03-28T02:08:49.264363+0000","last_unstale":"2024-03-28T02:10:12.392576+0000","last_undegraded":"2024-03-28T02:10:12.392576+0000","last_fullsized":"2024-03-28T02:10:12.392576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:14:09.178725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.395559+0000","last_change":"2024-03-28T02:08:49.261713+0000","last_active":"2024-03-28T02:10:12.395559+0000","last_peered":"2024-03-28T02:10:12.395559+0000","last_clean":"2024-03-28T02:10:12.395559+0000","last_became_active":"2024-03-28T02:08:49.261311+0000","last_became_peered":"2024-03-28T02:08:49.261311+0000","last_unstale":"2024-03-28T02:10:12.395559+0000","last_undegraded":"2024-03-28T02:10:12.395559+0000","last_fullsized":"2024-03-28T02:10:12.395559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:56:54.801575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.383712+0000","last_change":"2024-03-28T02:08:49.256674+0000","last_active":"2024-03-28T02:10:12.383712+0000","last_peered":"2024-03-28T02:10:12.383712+0000","last_clean":"2024-03-28T02:10:12.383712+0000","last_became_active":"2024-03-28T02:08:49.256360+0000","last_became_peered":"2024-03-28T02:08:49.256360+0000","last_unstale":"2024-03-28T02:10:12.383712+0000","last_undegraded":"2024-03-28T02:10:12.383712+0000","last_fullsized":"2024-03-28T02:10:12.383712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:44:14.817569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388927+0000","last_change":"2024-03-28T02:08:49.259829+0000","last_active":"2024-03-28T02:10:12.388927+0000","last_peered":"2024-03-28T02:10:12.388927+0000","last_clean":"2024-03-28T02:10:12.388927+0000","last_became_active":"2024-03-28T02:08:49.259419+0000","last_became_peered":"2024-03-28T02:08:49.259419+0000","last_unstale":"2024-03-28T02:10:12.388927+0000","last_undegraded":"2024-03-28T02:10:12.388927+0000","last_fullsized":"2024-03-28T02:10:12.388927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:49:46.023075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.398783+0000","last_change":"2024-03-28T02:08:49.263116+0000","last_active":"2024-03-28T02:10:12.398783+0000","last_peered":"2024-03-28T02:10:12.398783+0000","last_clean":"2024-03-28T02:10:12.398783+0000","last_became_active":"2024-03-28T02:08:49.262568+0000","last_became_peered":"2024-03-28T02:08:49.262568+0000","last_unstale":"2024-03-28T02:10:12.398783+0000","last_undegraded":"2024-03-28T02:10:12.398783+0000","last_fullsized":"2024-03-28T02:10:12.398783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:29:50.397737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.393017+0000","last_change":"2024-03-28T02:08:49.261449+0000","last_active":"2024-03-28T02:10:12.393017+0000","last_peered":"2024-03-28T02:10:12.393017+0000","last_clean":"2024-03-28T02:10:12.393017+0000","last_became_active":"2024-03-28T02:08:49.261092+0000","last_became_peered":"2024-03-28T02:08:49.261092+0000","last_unstale":"2024-03-28T02:10:12.393017+0000","last_undegraded":"2024-03-28T02:10:12.393017+0000","last_fullsized":"2024-03-28T02:10:12.393017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:51:39.388852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.387857+0000","last_change":"2024-03-28T02:08:49.260711+0000","last_active":"2024-03-28T02:10:12.387857+0000","last_peered":"2024-03-28T02:10:12.387857+0000","last_clean":"2024-03-28T02:10:12.387857+0000","last_became_active":"2024-03-28T02:08:49.259358+0000","last_became_peered":"2024-03-28T02:08:49.259358+0000","last_unstale":"2024-03-28T02:10:12.387857+0000","last_undegraded":"2024-03-28T02:10:12.387857+0000","last_fullsized":"2024-03-28T02:10:12.387857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:27:09.084143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390756+0000","last_change":"2024-03-28T02:08:49.267145+0000","last_active":"2024-03-28T02:10:12.390756+0000","last_peered":"2024-03-28T02:10:12.390756+0000","last_clean":"2024-03-28T02:10:12.390756+0000","last_became_active":"2024-03-28T02:08:49.266826+0000","last_became_peered":"2024-03-28T02:08:49.266826+0000","last_unstale":"2024-03-28T02:10:12.390756+0000","last_undegraded":"2024-03-28T02:10:12.390756+0000","last_fullsized":"2024-03-28T02:10:12.390756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:37:20.143596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.403822+0000","last_change":"2024-03-28T02:08:49.266752+0000","last_active":"2024-03-28T02:10:12.403822+0000","last_peered":"2024-03-28T02:10:12.403822+0000","last_clean":"2024-03-28T02:10:12.403822+0000","last_became_active":"2024-03-28T02:08:49.266436+0000","last_became_peered":"2024-03-28T02:08:49.266436+0000","last_unstale":"2024-03-28T02:10:12.403822+0000","last_undegraded":"2024-03-28T02:10:12.403822+0000","last_fullsized":"2024-03-28T02:10:12.403822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:52:26.578174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388673+0000","last_change":"2024-03-28T02:08:49.258934+0000","last_active":"2024-03-28T02:10:12.388673+0000","last_peered":"2024-03-28T02:10:12.388673+0000","last_clean":"2024-03-28T02:10:12.388673+0000","last_became_active":"2024-03-28T02:08:49.258403+0000","last_became_peered":"2024-03-28T02:08:49.258403+0000","last_unstale":"2024-03-28T02:10:12.388673+0000","last_undegraded":"2024-03-28T02:10:12.388673+0000","last_fullsized":"2024-03-28T02:10:12.388673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:44:45.886136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.391565+0000","last_change":"2024-03-28T02:08:49.268456+0000","last_active":"2024-03-28T02:10:12.391565+0000","last_peered":"2024-03-28T02:10:12.391565+0000","last_clean":"2024-03-28T02:10:12.391565+0000","last_became_active":"2024-03-28T02:08:49.267346+0000","last_became_peered":"2024-03-28T02:08:49.267346+0000","last_unstale":"2024-03-28T02:10:12.391565+0000","last_undegraded":"2024-03-28T02:10:12.391565+0000","last_fullsized":"2024-03-28T02:10:12.391565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:16:47.157938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283904,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73947,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":766,"num_read_kb":467,"num_write":246,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":180,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":221841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":501550,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1159807,"data_compressed":56741,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":501300,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":501566,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1098316,"data_compressed":56741,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496804,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":564827,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":498260,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":564979,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":496512,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":501410,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1085440,"data_stored":1093366,"data_compressed":56741,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T02:10:25.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T02:10:25.756 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:25.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:25 smithi106 ceph-mon[31965]: pgmap v342: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:25 smithi178 ceph-mon[36521]: pgmap v342: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:25 smithi153 ceph-mon[36228]: pgmap v342: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:10:26.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:26 smithi106 ceph-mon[31965]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:10:26.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:26 smithi178 ceph-mon[36521]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:10:26.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:26 smithi153 ceph-mon[36228]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.476+0000 7f8b5d568700 1 -- 172.21.15.106:0/3176724705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581081c0 msgr2=0x7f8b581085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.476+0000 7f8b5d568700 1 --2- 172.21.15.106:0/3176724705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581081c0 0x7f8b581085a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0099c0 tx=0x7f8b4c009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.476+0000 7f8b5d568700 1 -- 172.21.15.106:0/3176724705 shutdown_connections 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.476+0000 7f8b5d568700 1 --2- 172.21.15.106:0/3176724705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b58102ba0 0x7f8b5810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.476+0000 7f8b5d568700 1 --2- 172.21.15.106:0/3176724705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581081c0 0x7f8b581085a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.476+0000 7f8b5d568700 1 --2- 172.21.15.106:0/3176724705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b581021e0 0x7f8b58102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.476+0000 7f8b5d568700 1 -- 172.21.15.106:0/3176724705 >> 172.21.15.106:0/3176724705 conn(0x7f8b580792c0 msgr2=0x7f8b580796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.476+0000 7f8b5d568700 1 -- 172.21.15.106:0/3176724705 shutdown_connections 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.476+0000 7f8b5d568700 1 -- 172.21.15.106:0/3176724705 wait complete. 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.477+0000 7f8b5d568700 1 Processor -- start 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.477+0000 7f8b5d568700 1 -- start start 2024-03-28T02:10:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.477+0000 7f8b5d568700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581021e0 0x7f8b5819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.477+0000 7f8b5d568700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b58102ba0 0x7f8b5819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.477+0000 7f8b5d568700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b581081c0 0x7f8b581987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.477+0000 7f8b5d568700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b581124c0 con 0x7f8b581021e0 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.477+0000 7f8b5d568700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58112340 con 0x7f8b581081c0 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.477+0000 7f8b5d568700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58112640 con 0x7f8b58102ba0 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b56ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581021e0 0x7f8b5819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b56ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581021e0 0x7f8b5819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:52726/0 (socket says 172.21.15.106:52726) 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b56ffd700 1 -- 172.21.15.106:0/2007442670 learned_addr learned my addr 172.21.15.106:0/2007442670 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b567fc700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b58102ba0 0x7f8b5819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b577fe700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b581081c0 0x7f8b581987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b56ffd700 1 -- 172.21.15.106:0/2007442670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b58102ba0 msgr2=0x7f8b5819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b56ffd700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b58102ba0 0x7f8b5819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b56ffd700 1 -- 172.21.15.106:0/2007442670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b581081c0 msgr2=0x7f8b581987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:27.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b56ffd700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b581081c0 0x7f8b581987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b56ffd700 1 -- 172.21.15.106:0/2007442670 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b58198f80 con 0x7f8b581021e0 2024-03-28T02:10:27.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b567fc700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b58102ba0 0x7f8b5819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:27.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b56ffd700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581021e0 0x7f8b5819e620 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c005910 tx=0x7f8b4c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:27.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b3ffff700 1 -- 172.21.15.106:0/2007442670 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c027070 con 0x7f8b581021e0 2024-03-28T02:10:27.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b58199210 con 0x7f8b581021e0 2024-03-28T02:10:27.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b3ffff700 1 -- 172.21.15.106:0/2007442670 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b4c003680 con 0x7f8b581021e0 2024-03-28T02:10:27.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b3ffff700 1 -- 172.21.15.106:0/2007442670 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c0173f0 con 0x7f8b581021e0 2024-03-28T02:10:27.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.478+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b581a5480 con 0x7f8b581021e0 2024-03-28T02:10:27.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.479+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b58103870 con 0x7f8b581021e0 2024-03-28T02:10:27.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.480+0000 7f8b3ffff700 1 -- 172.21.15.106:0/2007442670 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8b4c022590 con 0x7f8b581021e0 2024-03-28T02:10:27.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.481+0000 7f8b3ffff700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8b400a2890 0x7f8b400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:27.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.481+0000 7f8b567fc700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8b400a2890 0x7f8b400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:27.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.481+0000 7f8b3ffff700 1 -- 172.21.15.106:0/2007442670 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f8b4c0cf7a0 con 0x7f8b581021e0 2024-03-28T02:10:27.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.481+0000 7f8b567fc700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8b400a2890 0x7f8b400a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b580798a0 tx=0x7f8b48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:27.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.486+0000 7f8b3ffff700 1 -- 172.21.15.106:0/2007442670 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8b4c013070 con 0x7f8b581021e0 2024-03-28T02:10:27.621 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:27 smithi178 ceph-mon[36521]: pgmap v343: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T02:10:27.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.678+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8b58199cd0 con 0x7f8b400a2890 2024-03-28T02:10:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.688+0000 7f8b3ffff700 1 -- 172.21.15.106:0/2007442670 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340814 (secure 0 0 0) 0x7f8b58199cd0 con 0x7f8b400a2890 2024-03-28T02:10:27.685 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8b400a2890 msgr2=0x7f8b400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8b400a2890 0x7f8b400a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b580798a0 tx=0x7f8b48009260 comp rx=0 tx=0).stop 2024-03-28T02:10:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581021e0 msgr2=0x7f8b5819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581021e0 0x7f8b5819e620 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c005910 tx=0x7f8b4c00f690 comp rx=0 tx=0).stop 2024-03-28T02:10:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 shutdown_connections 2024-03-28T02:10:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b581081c0 0x7f8b581987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8b400a2890 0x7f8b400a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b581021e0 0x7f8b5819e620 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 --2- 172.21.15.106:0/2007442670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b58102ba0 0x7f8b5819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:27.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 >> 172.21.15.106:0/2007442670 conn(0x7f8b580792c0 msgr2=0x7f8b58100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:27.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 shutdown_connections 2024-03-28T02:10:27.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:27.691+0000 7f8b5d568700 1 -- 172.21.15.106:0/2007442670 wait complete. 2024-03-28T02:10:27.688 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-03-28T02:10:27.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:27 smithi106 ceph-mon[31965]: pgmap v343: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T02:10:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:27 smithi153 ceph-mon[36228]: pgmap v343: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T02:10:28.485 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-03-28T02:10:26.378246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664549,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":927,"num_read_kb":591,"num_write":429,"num_write_kb":3086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1181033,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":854,"ondisk_log_size":854,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378112,"kb_used_data":12320,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563712,"statfs":{"total":1151940427776,"available":1151553241088,"internally_reserved":0,"allocated":12615680,"data_stored":7978697,"data_compressed":355965,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":55,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003154"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.397218+0000","last_change":"2024-03-28T02:08:49.268140+0000","last_active":"2024-03-28T02:10:12.397218+0000","last_peered":"2024-03-28T02:10:12.397218+0000","last_clean":"2024-03-28T02:10:12.397218+0000","last_became_active":"2024-03-28T02:08:49.267695+0000","last_became_peered":"2024-03-28T02:08:49.267695+0000","last_unstale":"2024-03-28T02:10:12.397218+0000","last_undegraded":"2024-03-28T02:10:12.397218+0000","last_fullsized":"2024-03-28T02:10:12.397218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:27:09.768237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.385752+0000","last_change":"2024-03-28T02:08:49.259914+0000","last_active":"2024-03-28T02:10:12.385752+0000","last_peered":"2024-03-28T02:10:12.385752+0000","last_clean":"2024-03-28T02:10:12.385752+0000","last_became_active":"2024-03-28T02:08:49.259579+0000","last_became_peered":"2024-03-28T02:08:49.259579+0000","last_unstale":"2024-03-28T02:10:12.385752+0000","last_undegraded":"2024-03-28T02:10:12.385752+0000","last_fullsized":"2024-03-28T02:10:12.385752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:04:02.664429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402092+0000","last_change":"2024-03-28T02:08:49.268234+0000","last_active":"2024-03-28T02:10:12.402092+0000","last_peered":"2024-03-28T02:10:12.402092+0000","last_clean":"2024-03-28T02:10:12.402092+0000","last_became_active":"2024-03-28T02:08:49.267813+0000","last_became_peered":"2024-03-28T02:08:49.267813+0000","last_unstale":"2024-03-28T02:10:12.402092+0000","last_undegraded":"2024-03-28T02:10:12.402092+0000","last_fullsized":"2024-03-28T02:10:12.402092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:46:21.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.395926+0000","last_change":"2024-03-28T02:08:49.268397+0000","last_active":"2024-03-28T02:10:12.395926+0000","last_peered":"2024-03-28T02:10:12.395926+0000","last_clean":"2024-03-28T02:10:12.395926+0000","last_became_active":"2024-03-28T02:08:49.268059+0000","last_became_peered":"2024-03-28T02:08:49.268059+0000","last_unstale":"2024-03-28T02:10:12.395926+0000","last_undegraded":"2024-03-28T02:10:12.395926+0000","last_fullsized":"2024-03-28T02:10:12.395926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:45:51.951306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.385728+0000","last_change":"2024-03-28T02:08:49.257827+0000","last_active":"2024-03-28T02:10:12.385728+0000","last_peered":"2024-03-28T02:10:12.385728+0000","last_clean":"2024-03-28T02:10:12.385728+0000","last_became_active":"2024-03-28T02:08:49.256707+0000","last_became_peered":"2024-03-28T02:08:49.256707+0000","last_unstale":"2024-03-28T02:10:12.385728+0000","last_undegraded":"2024-03-28T02:10:12.385728+0000","last_fullsized":"2024-03-28T02:10:12.385728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:23:13.617900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.384334+0000","last_change":"2024-03-28T02:08:49.262545+0000","last_active":"2024-03-28T02:10:12.384334+0000","last_peered":"2024-03-28T02:10:12.384334+0000","last_clean":"2024-03-28T02:10:12.384334+0000","last_became_active":"2024-03-28T02:08:49.262218+0000","last_became_peered":"2024-03-28T02:08:49.262218+0000","last_unstale":"2024-03-28T02:10:12.384334+0000","last_undegraded":"2024-03-28T02:10:12.384334+0000","last_fullsized":"2024-03-28T02:10:12.384334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:55:20.418608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.386346+0000","last_change":"2024-03-28T02:08:49.260072+0000","last_active":"2024-03-28T02:10:12.386346+0000","last_peered":"2024-03-28T02:10:12.386346+0000","last_clean":"2024-03-28T02:10:12.386346+0000","last_became_active":"2024-03-28T02:08:49.259724+0000","last_became_peered":"2024-03-28T02:08:49.259724+0000","last_unstale":"2024-03-28T02:10:12.386346+0000","last_undegraded":"2024-03-28T02:10:12.386346+0000","last_fullsized":"2024-03-28T02:10:12.386346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:05.445467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.387161+0000","last_change":"2024-03-28T02:08:49.262547+0000","last_active":"2024-03-28T02:10:12.387161+0000","last_peered":"2024-03-28T02:10:12.387161+0000","last_clean":"2024-03-28T02:10:12.387161+0000","last_became_active":"2024-03-28T02:08:49.261836+0000","last_became_peered":"2024-03-28T02:08:49.261836+0000","last_unstale":"2024-03-28T02:10:12.387161+0000","last_undegraded":"2024-03-28T02:10:12.387161+0000","last_fullsized":"2024-03-28T02:10:12.387161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:55:22.201524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.403467+0000","last_change":"2024-03-28T02:08:49.262725+0000","last_active":"2024-03-28T02:10:12.403467+0000","last_peered":"2024-03-28T02:10:12.403467+0000","last_clean":"2024-03-28T02:10:12.403467+0000","last_became_active":"2024-03-28T02:08:49.262319+0000","last_became_peered":"2024-03-28T02:08:49.262319+0000","last_unstale":"2024-03-28T02:10:12.403467+0000","last_undegraded":"2024-03-28T02:10:12.403467+0000","last_fullsized":"2024-03-28T02:10:12.403467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:56:24.904009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.389220+0000","last_change":"2024-03-28T02:08:49.262194+0000","last_active":"2024-03-28T02:10:12.389220+0000","last_peered":"2024-03-28T02:10:12.389220+0000","last_clean":"2024-03-28T02:10:12.389220+0000","last_became_active":"2024-03-28T02:08:49.261887+0000","last_became_peered":"2024-03-28T02:08:49.261887+0000","last_unstale":"2024-03-28T02:10:12.389220+0000","last_undegraded":"2024-03-28T02:10:12.389220+0000","last_fullsized":"2024-03-28T02:10:12.389220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:52:47.049430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402139+0000","last_change":"2024-03-28T02:08:49.269222+0000","last_active":"2024-03-28T02:10:12.402139+0000","last_peered":"2024-03-28T02:10:12.402139+0000","last_clean":"2024-03-28T02:10:12.402139+0000","last_became_active":"2024-03-28T02:08:49.268647+0000","last_became_peered":"2024-03-28T02:08:49.268647+0000","last_unstale":"2024-03-28T02:10:12.402139+0000","last_undegraded":"2024-03-28T02:10:12.402139+0000","last_fullsized":"2024-03-28T02:10:12.402139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:10:01.108158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.389425+0000","last_change":"2024-03-28T02:08:49.269868+0000","last_active":"2024-03-28T02:10:12.389425+0000","last_peered":"2024-03-28T02:10:12.389425+0000","last_clean":"2024-03-28T02:10:12.389425+0000","last_became_active":"2024-03-28T02:08:49.269035+0000","last_became_peered":"2024-03-28T02:08:49.269035+0000","last_unstale":"2024-03-28T02:10:12.389425+0000","last_undegraded":"2024-03-28T02:10:12.389425+0000","last_fullsized":"2024-03-28T02:10:12.389425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:31.153970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388726+0000","last_change":"2024-03-28T02:08:49.254162+0000","last_active":"2024-03-28T02:10:12.388726+0000","last_peered":"2024-03-28T02:10:12.388726+0000","last_clean":"2024-03-28T02:10:12.388726+0000","last_became_active":"2024-03-28T02:08:49.253756+0000","last_became_peered":"2024-03-28T02:08:49.253756+0000","last_unstale":"2024-03-28T02:10:12.388726+0000","last_undegraded":"2024-03-28T02:10:12.388726+0000","last_fullsized":"2024-03-28T02:10:12.388726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:22:20.791809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388266+0000","last_change":"2024-03-28T02:08:49.257940+0000","last_active":"2024-03-28T02:10:12.388266+0000","last_peered":"2024-03-28T02:10:12.388266+0000","last_clean":"2024-03-28T02:10:12.388266+0000","last_became_active":"2024-03-28T02:08:49.257039+0000","last_became_peered":"2024-03-28T02:08:49.257039+0000","last_unstale":"2024-03-28T02:10:12.388266+0000","last_undegraded":"2024-03-28T02:10:12.388266+0000","last_fullsized":"2024-03-28T02:10:12.388266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:36:10.701760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.393556+0000","last_change":"2024-03-28T02:08:49.267502+0000","last_active":"2024-03-28T02:10:12.393556+0000","last_peered":"2024-03-28T02:10:12.393556+0000","last_clean":"2024-03-28T02:10:12.393556+0000","last_became_active":"2024-03-28T02:08:49.265447+0000","last_became_peered":"2024-03-28T02:08:49.265447+0000","last_unstale":"2024-03-28T02:10:12.393556+0000","last_undegraded":"2024-03-28T02:10:12.393556+0000","last_fullsized":"2024-03-28T02:10:12.393556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:47:44.891440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.403441+0000","last_change":"2024-03-28T02:08:49.267702+0000","last_active":"2024-03-28T02:10:12.403441+0000","last_peered":"2024-03-28T02:10:12.403441+0000","last_clean":"2024-03-28T02:10:12.403441+0000","last_became_active":"2024-03-28T02:08:49.267381+0000","last_became_peered":"2024-03-28T02:08:49.267381+0000","last_unstale":"2024-03-28T02:10:12.403441+0000","last_undegraded":"2024-03-28T02:10:12.403441+0000","last_fullsized":"2024-03-28T02:10:12.403441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:55:59.904141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.397366+0000","last_change":"2024-03-28T02:08:49.268240+0000","last_active":"2024-03-28T02:10:12.397366+0000","last_peered":"2024-03-28T02:10:12.397366+0000","last_clean":"2024-03-28T02:10:12.397366+0000","last_became_active":"2024-03-28T02:08:49.267884+0000","last_became_peered":"2024-03-28T02:08:49.267884+0000","last_unstale":"2024-03-28T02:10:12.397366+0000","last_undegraded":"2024-03-28T02:10:12.397366+0000","last_fullsized":"2024-03-28T02:10:12.397366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:34:23.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.384295+0000","last_change":"2024-03-28T02:08:49.262585+0000","last_active":"2024-03-28T02:10:12.384295+0000","last_peered":"2024-03-28T02:10:12.384295+0000","last_clean":"2024-03-28T02:10:12.384295+0000","last_became_active":"2024-03-28T02:08:49.262217+0000","last_became_peered":"2024-03-28T02:08:49.262217+0000","last_unstale":"2024-03-28T02:10:12.384295+0000","last_undegraded":"2024-03-28T02:10:12.384295+0000","last_fullsized":"2024-03-28T02:10:12.384295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:10:05.714733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.392792+0000","last_change":"2024-03-28T02:08:49.261550+0000","last_active":"2024-03-28T02:10:12.392792+0000","last_peered":"2024-03-28T02:10:12.392792+0000","last_clean":"2024-03-28T02:10:12.392792+0000","last_became_active":"2024-03-28T02:08:49.261196+0000","last_became_peered":"2024-03-28T02:08:49.261196+0000","last_unstale":"2024-03-28T02:10:12.392792+0000","last_undegraded":"2024-03-28T02:10:12.392792+0000","last_fullsized":"2024-03-28T02:10:12.392792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:52:52.972048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457157+0000","last_change":"2024-03-28T02:08:53.491877+0000","last_active":"2024-03-28T02:09:43.457157+0000","last_peered":"2024-03-28T02:09:43.457157+0000","last_clean":"2024-03-28T02:09:43.457157+0000","last_became_active":"2024-03-28T02:08:53.491584+0000","last_became_peered":"2024-03-28T02:08:53.491584+0000","last_unstale":"2024-03-28T02:09:43.457157+0000","last_undegraded":"2024-03-28T02:09:43.457157+0000","last_fullsized":"2024-03-28T02:09:43.457157+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:47:41.778839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:22.380349+0000","last_change":"2024-03-28T02:08:48.448277+0000","last_active":"2024-03-28T02:10:22.380349+0000","last_peered":"2024-03-28T02:10:22.380349+0000","last_clean":"2024-03-28T02:10:22.380349+0000","last_became_active":"2024-03-28T02:08:48.447910+0000","last_became_peered":"2024-03-28T02:08:48.447910+0000","last_unstale":"2024-03-28T02:10:22.380349+0000","last_undegraded":"2024-03-28T02:10:22.380349+0000","last_fullsized":"2024-03-28T02:10:22.380349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:13:34.241952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390546+0000","last_change":"2024-03-28T02:08:49.258840+0000","last_active":"2024-03-28T02:10:12.390546+0000","last_peered":"2024-03-28T02:10:12.390546+0000","last_clean":"2024-03-28T02:10:12.390546+0000","last_became_active":"2024-03-28T02:08:49.258259+0000","last_became_peered":"2024-03-28T02:08:49.258259+0000","last_unstale":"2024-03-28T02:10:12.390546+0000","last_undegraded":"2024-03-28T02:10:12.390546+0000","last_fullsized":"2024-03-28T02:10:12.390546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:25:31.170413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459442+0000","last_change":"2024-03-28T02:08:53.496476+0000","last_active":"2024-03-28T02:09:43.459442+0000","last_peered":"2024-03-28T02:09:43.459442+0000","last_clean":"2024-03-28T02:09:43.459442+0000","last_became_active":"2024-03-28T02:08:53.496034+0000","last_became_peered":"2024-03-28T02:08:53.496034+0000","last_unstale":"2024-03-28T02:09:43.459442+0000","last_undegraded":"2024-03-28T02:09:43.459442+0000","last_fullsized":"2024-03-28T02:09:43.459442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:49:00.550877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.464780+0000","last_change":"2024-03-28T02:08:48.457960+0000","last_active":"2024-03-28T02:09:43.464780+0000","last_peered":"2024-03-28T02:09:43.464780+0000","last_clean":"2024-03-28T02:09:43.464780+0000","last_became_active":"2024-03-28T02:08:48.455528+0000","last_became_peered":"2024-03-28T02:08:48.455528+0000","last_unstale":"2024-03-28T02:09:43.464780+0000","last_undegraded":"2024-03-28T02:09:43.464780+0000","last_fullsized":"2024-03-28T02:09:43.464780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:40:37.830166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402071+0000","last_change":"2024-03-28T02:08:49.268390+0000","last_active":"2024-03-28T02:10:12.402071+0000","last_peered":"2024-03-28T02:10:12.402071+0000","last_clean":"2024-03-28T02:10:12.402071+0000","last_became_active":"2024-03-28T02:08:49.268025+0000","last_became_peered":"2024-03-28T02:08:49.268025+0000","last_unstale":"2024-03-28T02:10:12.402071+0000","last_undegraded":"2024-03-28T02:10:12.402071+0000","last_fullsized":"2024-03-28T02:10:12.402071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:40:07.752553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.465487+0000","last_change":"2024-03-28T02:09:43.465487+0000","last_active":"2024-03-28T02:09:43.465487+0000","last_peered":"2024-03-28T02:09:43.465487+0000","last_clean":"2024-03-28T02:09:43.465487+0000","last_became_active":"2024-03-28T02:09:43.465015+0000","last_became_peered":"2024-03-28T02:09:43.465015+0000","last_unstale":"2024-03-28T02:09:43.465487+0000","last_undegraded":"2024-03-28T02:09:43.465487+0000","last_fullsized":"2024-03-28T02:09:43.465487+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:48:58.978284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,8],"acting":[4,3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461836+0000","last_change":"2024-03-28T02:08:48.459667+0000","last_active":"2024-03-28T02:09:43.461836+0000","last_peered":"2024-03-28T02:09:43.461836+0000","last_clean":"2024-03-28T02:09:43.461836+0000","last_became_active":"2024-03-28T02:08:48.459310+0000","last_became_peered":"2024-03-28T02:08:48.459310+0000","last_unstale":"2024-03-28T02:09:43.461836+0000","last_undegraded":"2024-03-28T02:09:43.461836+0000","last_fullsized":"2024-03-28T02:09:43.461836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:51:07.351151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.403849+0000","last_change":"2024-03-28T02:08:49.254576+0000","last_active":"2024-03-28T02:10:12.403849+0000","last_peered":"2024-03-28T02:10:12.403849+0000","last_clean":"2024-03-28T02:10:12.403849+0000","last_became_active":"2024-03-28T02:08:49.254213+0000","last_became_peered":"2024-03-28T02:08:49.254213+0000","last_unstale":"2024-03-28T02:10:12.403849+0000","last_undegraded":"2024-03-28T02:10:12.403849+0000","last_fullsized":"2024-03-28T02:10:12.403849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.630801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459713+0000","last_change":"2024-03-28T02:08:53.493624+0000","last_active":"2024-03-28T02:09:43.459713+0000","last_peered":"2024-03-28T02:09:43.459713+0000","last_clean":"2024-03-28T02:09:43.459713+0000","last_became_active":"2024-03-28T02:08:53.493195+0000","last_became_peered":"2024-03-28T02:08:53.493195+0000","last_unstale":"2024-03-28T02:09:43.459713+0000","last_undegraded":"2024-03-28T02:09:43.459713+0000","last_fullsized":"2024-03-28T02:09:43.459713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:38:25.497464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.462020+0000","last_change":"2024-03-28T02:08:48.454008+0000","last_active":"2024-03-28T02:09:43.462020+0000","last_peered":"2024-03-28T02:09:43.462020+0000","last_clean":"2024-03-28T02:09:43.462020+0000","last_became_active":"2024-03-28T02:08:48.450674+0000","last_became_peered":"2024-03-28T02:08:48.450674+0000","last_unstale":"2024-03-28T02:09:43.462020+0000","last_undegraded":"2024-03-28T02:09:43.462020+0000","last_fullsized":"2024-03-28T02:09:43.462020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:08:12.902348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402185+0000","last_change":"2024-03-28T02:08:49.266552+0000","last_active":"2024-03-28T02:10:12.402185+0000","last_peered":"2024-03-28T02:10:12.402185+0000","last_clean":"2024-03-28T02:10:12.402185+0000","last_became_active":"2024-03-28T02:08:49.265161+0000","last_became_peered":"2024-03-28T02:08:49.265161+0000","last_unstale":"2024-03-28T02:10:12.402185+0000","last_undegraded":"2024-03-28T02:10:12.402185+0000","last_fullsized":"2024-03-28T02:10:12.402185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:22.128908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457357+0000","last_change":"2024-03-28T02:08:53.487871+0000","last_active":"2024-03-28T02:09:43.457357+0000","last_peered":"2024-03-28T02:09:43.457357+0000","last_clean":"2024-03-28T02:09:43.457357+0000","last_became_active":"2024-03-28T02:08:53.487344+0000","last_became_peered":"2024-03-28T02:08:53.487344+0000","last_unstale":"2024-03-28T02:09:43.457357+0000","last_undegraded":"2024-03-28T02:09:43.457357+0000","last_fullsized":"2024-03-28T02:09:43.457357+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:34:12.555549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:24.331062+0000","last_change":"2024-03-28T02:08:48.450507+0000","last_active":"2024-03-28T02:10:24.331062+0000","last_peered":"2024-03-28T02:10:24.331062+0000","last_clean":"2024-03-28T02:10:24.331062+0000","last_became_active":"2024-03-28T02:08:48.450065+0000","last_became_peered":"2024-03-28T02:08:48.450065+0000","last_unstale":"2024-03-28T02:10:24.331062+0000","last_undegraded":"2024-03-28T02:10:24.331062+0000","last_fullsized":"2024-03-28T02:10:24.331062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:00:45.393364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.383059+0000","last_change":"2024-03-28T02:08:49.256313+0000","last_active":"2024-03-28T02:10:12.383059+0000","last_peered":"2024-03-28T02:10:12.383059+0000","last_clean":"2024-03-28T02:10:12.383059+0000","last_became_active":"2024-03-28T02:08:49.255957+0000","last_became_peered":"2024-03-28T02:08:49.255957+0000","last_unstale":"2024-03-28T02:10:12.383059+0000","last_undegraded":"2024-03-28T02:10:12.383059+0000","last_fullsized":"2024-03-28T02:10:12.383059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:39:27.402983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457298+0000","last_change":"2024-03-28T02:08:53.498554+0000","last_active":"2024-03-28T02:09:43.457298+0000","last_peered":"2024-03-28T02:09:43.457298+0000","last_clean":"2024-03-28T02:09:43.457298+0000","last_became_active":"2024-03-28T02:08:53.498152+0000","last_became_peered":"2024-03-28T02:08:53.498152+0000","last_unstale":"2024-03-28T02:09:43.457298+0000","last_undegraded":"2024-03-28T02:09:43.457298+0000","last_fullsized":"2024-03-28T02:09:43.457298+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:41:38.234758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.463720+0000","last_change":"2024-03-28T02:08:48.450494+0000","last_active":"2024-03-28T02:09:43.463720+0000","last_peered":"2024-03-28T02:09:43.463720+0000","last_clean":"2024-03-28T02:09:43.463720+0000","last_became_active":"2024-03-28T02:08:48.450158+0000","last_became_peered":"2024-03-28T02:08:48.450158+0000","last_unstale":"2024-03-28T02:09:43.463720+0000","last_undegraded":"2024-03-28T02:09:43.463720+0000","last_fullsized":"2024-03-28T02:09:43.463720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:47:20.021116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.377248+0000","last_change":"2024-03-28T02:09:43.465511+0000","last_active":"2024-03-28T02:10:12.377248+0000","last_peered":"2024-03-28T02:10:12.377248+0000","last_clean":"2024-03-28T02:10:12.377248+0000","last_became_active":"2024-03-28T02:09:43.465096+0000","last_became_peered":"2024-03-28T02:09:43.465096+0000","last_unstale":"2024-03-28T02:10:12.377248+0000","last_undegraded":"2024-03-28T02:10:12.377248+0000","last_fullsized":"2024-03-28T02:10:12.377248+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:37.879449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460319+0000","last_change":"2024-03-28T02:08:53.490046+0000","last_active":"2024-03-28T02:09:43.460319+0000","last_peered":"2024-03-28T02:09:43.460319+0000","last_clean":"2024-03-28T02:09:43.460319+0000","last_became_active":"2024-03-28T02:08:53.489463+0000","last_became_peered":"2024-03-28T02:08:53.489463+0000","last_unstale":"2024-03-28T02:09:43.460319+0000","last_undegraded":"2024-03-28T02:09:43.460319+0000","last_fullsized":"2024-03-28T02:09:43.460319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:25:03.485568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459782+0000","last_change":"2024-03-28T02:08:48.450548+0000","last_active":"2024-03-28T02:09:43.459782+0000","last_peered":"2024-03-28T02:09:43.459782+0000","last_clean":"2024-03-28T02:09:43.459782+0000","last_became_active":"2024-03-28T02:08:48.450068+0000","last_became_peered":"2024-03-28T02:08:48.450068+0000","last_unstale":"2024-03-28T02:09:43.459782+0000","last_undegraded":"2024-03-28T02:09:43.459782+0000","last_fullsized":"2024-03-28T02:09:43.459782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:24:05.231774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.381877+0000","last_change":"2024-03-28T02:08:49.262785+0000","last_active":"2024-03-28T02:10:12.381877+0000","last_peered":"2024-03-28T02:10:12.381877+0000","last_clean":"2024-03-28T02:10:12.381877+0000","last_became_active":"2024-03-28T02:08:49.262253+0000","last_became_peered":"2024-03-28T02:08:49.262253+0000","last_unstale":"2024-03-28T02:10:12.381877+0000","last_undegraded":"2024-03-28T02:10:12.381877+0000","last_fullsized":"2024-03-28T02:10:12.381877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:07:37.064693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.464151+0000","last_change":"2024-03-28T02:08:53.483738+0000","last_active":"2024-03-28T02:09:43.464151+0000","last_peered":"2024-03-28T02:09:43.464151+0000","last_clean":"2024-03-28T02:09:43.464151+0000","last_became_active":"2024-03-28T02:08:53.483214+0000","last_became_peered":"2024-03-28T02:08:53.483214+0000","last_unstale":"2024-03-28T02:09:43.464151+0000","last_undegraded":"2024-03-28T02:09:43.464151+0000","last_fullsized":"2024-03-28T02:09:43.464151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:06:32.789839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461333+0000","last_change":"2024-03-28T02:08:48.448858+0000","last_active":"2024-03-28T02:09:43.461333+0000","last_peered":"2024-03-28T02:09:43.461333+0000","last_clean":"2024-03-28T02:09:43.461333+0000","last_became_active":"2024-03-28T02:08:48.448435+0000","last_became_peered":"2024-03-28T02:08:48.448435+0000","last_unstale":"2024-03-28T02:09:43.461333+0000","last_undegraded":"2024-03-28T02:09:43.461333+0000","last_fullsized":"2024-03-28T02:09:43.461333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:34:13.035336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.380421+0000","last_change":"2024-03-28T02:08:49.261354+0000","last_active":"2024-03-28T02:10:12.380421+0000","last_peered":"2024-03-28T02:10:12.380421+0000","last_clean":"2024-03-28T02:10:12.380421+0000","last_became_active":"2024-03-28T02:08:49.261038+0000","last_became_peered":"2024-03-28T02:08:49.261038+0000","last_unstale":"2024-03-28T02:10:12.380421+0000","last_undegraded":"2024-03-28T02:10:12.380421+0000","last_fullsized":"2024-03-28T02:10:12.380421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:03:15.615484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.464323+0000","last_change":"2024-03-28T02:08:53.488573+0000","last_active":"2024-03-28T02:09:43.464323+0000","last_peered":"2024-03-28T02:09:43.464323+0000","last_clean":"2024-03-28T02:09:43.464323+0000","last_became_active":"2024-03-28T02:08:53.488198+0000","last_became_peered":"2024-03-28T02:08:53.488198+0000","last_unstale":"2024-03-28T02:09:43.464323+0000","last_undegraded":"2024-03-28T02:09:43.464323+0000","last_fullsized":"2024-03-28T02:09:43.464323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:43:29.158223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":179,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:23.329708+0000","last_change":"2024-03-28T02:08:48.445941+0000","last_active":"2024-03-28T02:10:23.329708+0000","last_peered":"2024-03-28T02:10:23.329708+0000","last_clean":"2024-03-28T02:10:23.329708+0000","last_became_active":"2024-03-28T02:08:48.445599+0000","last_became_peered":"2024-03-28T02:08:48.445599+0000","last_unstale":"2024-03-28T02:10:23.329708+0000","last_undegraded":"2024-03-28T02:10:23.329708+0000","last_fullsized":"2024-03-28T02:10:23.329708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:28:53.524708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.401651+0000","last_change":"2024-03-28T02:08:49.258592+0000","last_active":"2024-03-28T02:10:12.401651+0000","last_peered":"2024-03-28T02:10:12.401651+0000","last_clean":"2024-03-28T02:10:12.401651+0000","last_became_active":"2024-03-28T02:08:49.256527+0000","last_became_peered":"2024-03-28T02:08:49.256527+0000","last_unstale":"2024-03-28T02:10:12.401651+0000","last_undegraded":"2024-03-28T02:10:12.401651+0000","last_fullsized":"2024-03-28T02:10:12.401651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:54:41.645210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457589+0000","last_change":"2024-03-28T02:08:53.494677+0000","last_active":"2024-03-28T02:09:43.457589+0000","last_peered":"2024-03-28T02:09:43.457589+0000","last_clean":"2024-03-28T02:09:43.457589+0000","last_became_active":"2024-03-28T02:08:53.493076+0000","last_became_peered":"2024-03-28T02:08:53.493076+0000","last_unstale":"2024-03-28T02:09:43.457589+0000","last_undegraded":"2024-03-28T02:09:43.457589+0000","last_fullsized":"2024-03-28T02:09:43.457589+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:10:44.817473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461091+0000","last_change":"2024-03-28T02:08:48.454321+0000","last_active":"2024-03-28T02:09:43.461091+0000","last_peered":"2024-03-28T02:09:43.461091+0000","last_clean":"2024-03-28T02:09:43.461091+0000","last_became_active":"2024-03-28T02:08:48.453907+0000","last_became_peered":"2024-03-28T02:08:48.453907+0000","last_unstale":"2024-03-28T02:09:43.461091+0000","last_undegraded":"2024-03-28T02:09:43.461091+0000","last_fullsized":"2024-03-28T02:09:43.461091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:45:18.347073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.395714+0000","last_change":"2024-03-28T02:08:49.262196+0000","last_active":"2024-03-28T02:10:12.395714+0000","last_peered":"2024-03-28T02:10:12.395714+0000","last_clean":"2024-03-28T02:10:12.395714+0000","last_became_active":"2024-03-28T02:08:49.261190+0000","last_became_peered":"2024-03-28T02:08:49.261190+0000","last_unstale":"2024-03-28T02:10:12.395714+0000","last_undegraded":"2024-03-28T02:10:12.395714+0000","last_fullsized":"2024-03-28T02:10:12.395714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:37:15.939535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460131+0000","last_change":"2024-03-28T02:08:53.492207+0000","last_active":"2024-03-28T02:09:43.460131+0000","last_peered":"2024-03-28T02:09:43.460131+0000","last_clean":"2024-03-28T02:09:43.460131+0000","last_became_active":"2024-03-28T02:08:53.491831+0000","last_became_peered":"2024-03-28T02:08:53.491831+0000","last_unstale":"2024-03-28T02:09:43.460131+0000","last_undegraded":"2024-03-28T02:09:43.460131+0000","last_fullsized":"2024-03-28T02:09:43.460131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:00.163397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460597+0000","last_change":"2024-03-28T02:08:48.450908+0000","last_active":"2024-03-28T02:09:43.460597+0000","last_peered":"2024-03-28T02:09:43.460597+0000","last_clean":"2024-03-28T02:09:43.460597+0000","last_became_active":"2024-03-28T02:08:48.450382+0000","last_became_peered":"2024-03-28T02:08:48.450382+0000","last_unstale":"2024-03-28T02:09:43.460597+0000","last_undegraded":"2024-03-28T02:09:43.460597+0000","last_fullsized":"2024-03-28T02:09:43.460597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:26:25.851416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.404028+0000","last_change":"2024-03-28T02:08:49.268904+0000","last_active":"2024-03-28T02:10:12.404028+0000","last_peered":"2024-03-28T02:10:12.404028+0000","last_clean":"2024-03-28T02:10:12.404028+0000","last_became_active":"2024-03-28T02:08:49.268595+0000","last_became_peered":"2024-03-28T02:08:49.268595+0000","last_unstale":"2024-03-28T02:10:12.404028+0000","last_undegraded":"2024-03-28T02:10:12.404028+0000","last_fullsized":"2024-03-28T02:10:12.404028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:12:25.903861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457690+0000","last_change":"2024-03-28T02:08:53.489115+0000","last_active":"2024-03-28T02:09:43.457690+0000","last_peered":"2024-03-28T02:09:43.457690+0000","last_clean":"2024-03-28T02:09:43.457690+0000","last_became_active":"2024-03-28T02:08:53.488716+0000","last_became_peered":"2024-03-28T02:08:53.488716+0000","last_unstale":"2024-03-28T02:09:43.457690+0000","last_undegraded":"2024-03-28T02:09:43.457690+0000","last_fullsized":"2024-03-28T02:09:43.457690+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:00:42.920205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":225,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:23.329553+0000","last_change":"2024-03-28T02:08:48.453921+0000","last_active":"2024-03-28T02:10:23.329553+0000","last_peered":"2024-03-28T02:10:23.329553+0000","last_clean":"2024-03-28T02:10:23.329553+0000","last_became_active":"2024-03-28T02:08:48.453301+0000","last_became_peered":"2024-03-28T02:08:48.453301+0000","last_unstale":"2024-03-28T02:10:23.329553+0000","last_undegraded":"2024-03-28T02:10:23.329553+0000","last_fullsized":"2024-03-28T02:10:23.329553+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:57:04.543444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66993,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":161,"num_read_kb":145,"num_write":40,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.382903+0000","last_change":"2024-03-28T02:08:49.259942+0000","last_active":"2024-03-28T02:10:12.382903+0000","last_peered":"2024-03-28T02:10:12.382903+0000","last_clean":"2024-03-28T02:10:12.382903+0000","last_became_active":"2024-03-28T02:08:49.259602+0000","last_became_peered":"2024-03-28T02:08:49.259602+0000","last_unstale":"2024-03-28T02:10:12.382903+0000","last_undegraded":"2024-03-28T02:10:12.382903+0000","last_fullsized":"2024-03-28T02:10:12.382903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:38:05.897720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461108+0000","last_change":"2024-03-28T02:08:53.495675+0000","last_active":"2024-03-28T02:09:43.461108+0000","last_peered":"2024-03-28T02:09:43.461108+0000","last_clean":"2024-03-28T02:09:43.461108+0000","last_became_active":"2024-03-28T02:08:53.495233+0000","last_became_peered":"2024-03-28T02:08:53.495233+0000","last_unstale":"2024-03-28T02:09:43.461108+0000","last_undegraded":"2024-03-28T02:09:43.461108+0000","last_fullsized":"2024-03-28T02:09:43.461108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:43.805151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.488362+0000","last_change":"2024-03-28T02:09:43.488079+0000","last_active":"2024-03-28T02:09:43.488362+0000","last_peered":"2024-03-28T02:09:43.488362+0000","last_clean":"2024-03-28T02:09:43.488362+0000","last_became_active":"2024-03-28T02:09:43.467010+0000","last_became_peered":"2024-03-28T02:09:43.467010+0000","last_unstale":"2024-03-28T02:09:43.488362+0000","last_undegraded":"2024-03-28T02:09:43.488362+0000","last_fullsized":"2024-03-28T02:09:43.488362+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:47:56.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388205+0000","last_change":"2024-03-28T02:08:49.259179+0000","last_active":"2024-03-28T02:10:12.388205+0000","last_peered":"2024-03-28T02:10:12.388205+0000","last_clean":"2024-03-28T02:10:12.388205+0000","last_became_active":"2024-03-28T02:08:49.258538+0000","last_became_peered":"2024-03-28T02:08:49.258538+0000","last_unstale":"2024-03-28T02:10:12.388205+0000","last_undegraded":"2024-03-28T02:10:12.388205+0000","last_fullsized":"2024-03-28T02:10:12.388205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:12:20.008528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.458972+0000","last_change":"2024-03-28T02:08:53.499523+0000","last_active":"2024-03-28T02:09:43.458972+0000","last_peered":"2024-03-28T02:09:43.458972+0000","last_clean":"2024-03-28T02:09:43.458972+0000","last_became_active":"2024-03-28T02:08:53.499112+0000","last_became_peered":"2024-03-28T02:08:53.499112+0000","last_unstale":"2024-03-28T02:09:43.458972+0000","last_undegraded":"2024-03-28T02:09:43.458972+0000","last_fullsized":"2024-03-28T02:09:43.458972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:31:12.008640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459676+0000","last_change":"2024-03-28T02:08:48.452952+0000","last_active":"2024-03-28T02:09:43.459676+0000","last_peered":"2024-03-28T02:09:43.459676+0000","last_clean":"2024-03-28T02:09:43.459676+0000","last_became_active":"2024-03-28T02:08:48.452586+0000","last_became_peered":"2024-03-28T02:08:48.452586+0000","last_unstale":"2024-03-28T02:09:43.459676+0000","last_undegraded":"2024-03-28T02:09:43.459676+0000","last_fullsized":"2024-03-28T02:09:43.459676+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:44.749902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.391810+0000","last_change":"2024-03-28T02:08:49.256746+0000","last_active":"2024-03-28T02:10:12.391810+0000","last_peered":"2024-03-28T02:10:12.391810+0000","last_clean":"2024-03-28T02:10:12.391810+0000","last_became_active":"2024-03-28T02:08:49.256398+0000","last_became_peered":"2024-03-28T02:08:49.256398+0000","last_unstale":"2024-03-28T02:10:12.391810+0000","last_undegraded":"2024-03-28T02:10:12.391810+0000","last_fullsized":"2024-03-28T02:10:12.391810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:10:52.021218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.458352+0000","last_change":"2024-03-28T02:08:53.494257+0000","last_active":"2024-03-28T02:09:43.458352+0000","last_peered":"2024-03-28T02:09:43.458352+0000","last_clean":"2024-03-28T02:09:43.458352+0000","last_became_active":"2024-03-28T02:08:53.493889+0000","last_became_peered":"2024-03-28T02:08:53.493889+0000","last_unstale":"2024-03-28T02:09:43.458352+0000","last_undegraded":"2024-03-28T02:09:43.458352+0000","last_fullsized":"2024-03-28T02:09:43.458352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:49.292686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:24.382439+0000","last_change":"2024-03-28T02:08:48.447361+0000","last_active":"2024-03-28T02:10:24.382439+0000","last_peered":"2024-03-28T02:10:24.382439+0000","last_clean":"2024-03-28T02:10:24.382439+0000","last_became_active":"2024-03-28T02:08:48.446943+0000","last_became_peered":"2024-03-28T02:08:48.446943+0000","last_unstale":"2024-03-28T02:10:24.382439+0000","last_undegraded":"2024-03-28T02:10:24.382439+0000","last_fullsized":"2024-03-28T02:10:24.382439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:19:46.552447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":79,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388883+0000","last_change":"2024-03-28T02:08:49.255253+0000","last_active":"2024-03-28T02:10:12.388883+0000","last_peered":"2024-03-28T02:10:12.388883+0000","last_clean":"2024-03-28T02:10:12.388883+0000","last_became_active":"2024-03-28T02:08:49.254804+0000","last_became_peered":"2024-03-28T02:08:49.254804+0000","last_unstale":"2024-03-28T02:10:12.388883+0000","last_undegraded":"2024-03-28T02:10:12.388883+0000","last_fullsized":"2024-03-28T02:10:12.388883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:53:31.927830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.466912+0000","last_change":"2024-03-28T02:09:43.466912+0000","last_active":"2024-03-28T02:09:43.466912+0000","last_peered":"2024-03-28T02:09:43.466912+0000","last_clean":"2024-03-28T02:09:43.466912+0000","last_became_active":"2024-03-28T02:09:43.466462+0000","last_became_peered":"2024-03-28T02:09:43.466462+0000","last_unstale":"2024-03-28T02:09:43.466912+0000","last_undegraded":"2024-03-28T02:09:43.466912+0000","last_fullsized":"2024-03-28T02:09:43.466912+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:51:20.559601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.480457+0000","last_change":"2024-03-28T02:09:43.480082+0000","last_active":"2024-03-28T02:09:43.480457+0000","last_peered":"2024-03-28T02:09:43.480457+0000","last_clean":"2024-03-28T02:09:43.480457+0000","last_became_active":"2024-03-28T02:09:43.464495+0000","last_became_peered":"2024-03-28T02:09:43.464495+0000","last_unstale":"2024-03-28T02:09:43.480457+0000","last_undegraded":"2024-03-28T02:09:43.480457+0000","last_fullsized":"2024-03-28T02:09:43.480457+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:55:11.451630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.401459+0000","last_change":"2024-03-28T02:09:43.464753+0000","last_active":"2024-03-28T02:10:12.401459+0000","last_peered":"2024-03-28T02:10:12.401459+0000","last_clean":"2024-03-28T02:10:12.401459+0000","last_became_active":"2024-03-28T02:09:43.464400+0000","last_became_peered":"2024-03-28T02:09:43.464400+0000","last_unstale":"2024-03-28T02:10:12.401459+0000","last_undegraded":"2024-03-28T02:10:12.401459+0000","last_fullsized":"2024-03-28T02:10:12.401459+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:19:35.611109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460947+0000","last_change":"2024-03-28T02:08:53.490855+0000","last_active":"2024-03-28T02:09:43.460947+0000","last_peered":"2024-03-28T02:09:43.460947+0000","last_clean":"2024-03-28T02:09:43.460947+0000","last_became_active":"2024-03-28T02:08:53.490415+0000","last_became_peered":"2024-03-28T02:08:53.490415+0000","last_unstale":"2024-03-28T02:09:43.460947+0000","last_undegraded":"2024-03-28T02:09:43.460947+0000","last_fullsized":"2024-03-28T02:09:43.460947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:56:37.426398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.463168+0000","last_change":"2024-03-28T02:08:48.448909+0000","last_active":"2024-03-28T02:09:43.463168+0000","last_peered":"2024-03-28T02:09:43.463168+0000","last_clean":"2024-03-28T02:09:43.463168+0000","last_became_active":"2024-03-28T02:08:48.448548+0000","last_became_peered":"2024-03-28T02:08:48.448548+0000","last_unstale":"2024-03-28T02:09:43.463168+0000","last_undegraded":"2024-03-28T02:09:43.463168+0000","last_fullsized":"2024-03-28T02:09:43.463168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:51:06.206266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390639+0000","last_change":"2024-03-28T02:08:49.266920+0000","last_active":"2024-03-28T02:10:12.390639+0000","last_peered":"2024-03-28T02:10:12.390639+0000","last_clean":"2024-03-28T02:10:12.390639+0000","last_became_active":"2024-03-28T02:08:49.266600+0000","last_became_peered":"2024-03-28T02:08:49.266600+0000","last_unstale":"2024-03-28T02:10:12.390639+0000","last_undegraded":"2024-03-28T02:10:12.390639+0000","last_fullsized":"2024-03-28T02:10:12.390639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:57.735460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459737+0000","last_change":"2024-03-28T02:08:53.631636+0000","last_active":"2024-03-28T02:09:43.459737+0000","last_peered":"2024-03-28T02:09:43.459737+0000","last_clean":"2024-03-28T02:09:43.459737+0000","last_became_active":"2024-03-28T02:08:53.631183+0000","last_became_peered":"2024-03-28T02:08:53.631183+0000","last_unstale":"2024-03-28T02:09:43.459737+0000","last_undegraded":"2024-03-28T02:09:43.459737+0000","last_fullsized":"2024-03-28T02:09:43.459737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:22:09.212726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.458121+0000","last_change":"2024-03-28T02:08:48.440789+0000","last_active":"2024-03-28T02:09:43.458121+0000","last_peered":"2024-03-28T02:09:43.458121+0000","last_clean":"2024-03-28T02:09:43.458121+0000","last_became_active":"2024-03-28T02:08:48.440403+0000","last_became_peered":"2024-03-28T02:08:48.440403+0000","last_unstale":"2024-03-28T02:09:43.458121+0000","last_undegraded":"2024-03-28T02:09:43.458121+0000","last_fullsized":"2024-03-28T02:09:43.458121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:37.227279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.378816+0000","last_change":"2024-03-28T02:08:49.255340+0000","last_active":"2024-03-28T02:10:12.378816+0000","last_peered":"2024-03-28T02:10:12.378816+0000","last_clean":"2024-03-28T02:10:12.378816+0000","last_became_active":"2024-03-28T02:08:49.255027+0000","last_became_peered":"2024-03-28T02:08:49.255027+0000","last_unstale":"2024-03-28T02:10:12.378816+0000","last_undegraded":"2024-03-28T02:10:12.378816+0000","last_fullsized":"2024-03-28T02:10:12.378816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:54:03.454022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460139+0000","last_change":"2024-03-28T02:08:53.499721+0000","last_active":"2024-03-28T02:09:43.460139+0000","last_peered":"2024-03-28T02:09:43.460139+0000","last_clean":"2024-03-28T02:09:43.460139+0000","last_became_active":"2024-03-28T02:08:53.499117+0000","last_became_peered":"2024-03-28T02:08:53.499117+0000","last_unstale":"2024-03-28T02:09:43.460139+0000","last_undegraded":"2024-03-28T02:09:43.460139+0000","last_fullsized":"2024-03-28T02:09:43.460139+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:07:33.561249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:24.382725+0000","last_change":"2024-03-28T02:08:48.453592+0000","last_active":"2024-03-28T02:10:24.382725+0000","last_peered":"2024-03-28T02:10:24.382725+0000","last_clean":"2024-03-28T02:10:24.382725+0000","last_became_active":"2024-03-28T02:08:48.453169+0000","last_became_peered":"2024-03-28T02:08:48.453169+0000","last_unstale":"2024-03-28T02:10:24.382725+0000","last_undegraded":"2024-03-28T02:10:24.382725+0000","last_fullsized":"2024-03-28T02:10:24.382725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:33.163170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390591+0000","last_change":"2024-03-28T02:08:49.263143+0000","last_active":"2024-03-28T02:10:12.390591+0000","last_peered":"2024-03-28T02:10:12.390591+0000","last_clean":"2024-03-28T02:10:12.390591+0000","last_became_active":"2024-03-28T02:08:49.262829+0000","last_became_peered":"2024-03-28T02:08:49.262829+0000","last_unstale":"2024-03-28T02:10:12.390591+0000","last_undegraded":"2024-03-28T02:10:12.390591+0000","last_fullsized":"2024-03-28T02:10:12.390591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:01:07.809685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460229+0000","last_change":"2024-03-28T02:08:53.631595+0000","last_active":"2024-03-28T02:09:43.460229+0000","last_peered":"2024-03-28T02:09:43.460229+0000","last_clean":"2024-03-28T02:09:43.460229+0000","last_became_active":"2024-03-28T02:08:53.630863+0000","last_became_peered":"2024-03-28T02:08:53.630863+0000","last_unstale":"2024-03-28T02:09:43.460229+0000","last_undegraded":"2024-03-28T02:09:43.460229+0000","last_fullsized":"2024-03-28T02:09:43.460229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:38:44.712728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390396+0000","last_change":"2024-03-28T02:08:49.263523+0000","last_active":"2024-03-28T02:10:12.390396+0000","last_peered":"2024-03-28T02:10:12.390396+0000","last_clean":"2024-03-28T02:10:12.390396+0000","last_became_active":"2024-03-28T02:08:49.263079+0000","last_became_peered":"2024-03-28T02:08:49.263079+0000","last_unstale":"2024-03-28T02:10:12.390396+0000","last_undegraded":"2024-03-28T02:10:12.390396+0000","last_fullsized":"2024-03-28T02:10:12.390396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:15:37.516032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:24.382039+0000","last_change":"2024-03-28T02:08:48.455723+0000","last_active":"2024-03-28T02:10:24.382039+0000","last_peered":"2024-03-28T02:10:24.382039+0000","last_clean":"2024-03-28T02:10:24.382039+0000","last_became_active":"2024-03-28T02:08:48.455340+0000","last_became_peered":"2024-03-28T02:08:48.455340+0000","last_unstale":"2024-03-28T02:10:24.382039+0000","last_undegraded":"2024-03-28T02:10:24.382039+0000","last_fullsized":"2024-03-28T02:10:24.382039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:15:36.376221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.470892+0000","last_change":"2024-03-28T02:09:43.470892+0000","last_active":"2024-03-28T02:09:43.470892+0000","last_peered":"2024-03-28T02:09:43.470892+0000","last_clean":"2024-03-28T02:09:43.470892+0000","last_became_active":"2024-03-28T02:09:43.465112+0000","last_became_peered":"2024-03-28T02:09:43.465112+0000","last_unstale":"2024-03-28T02:09:43.470892+0000","last_undegraded":"2024-03-28T02:09:43.470892+0000","last_fullsized":"2024-03-28T02:09:43.470892+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:59:02.285845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.384529+0000","last_change":"2024-03-28T02:08:49.260380+0000","last_active":"2024-03-28T02:10:12.384529+0000","last_peered":"2024-03-28T02:10:12.384529+0000","last_clean":"2024-03-28T02:10:12.384529+0000","last_became_active":"2024-03-28T02:08:49.260012+0000","last_became_peered":"2024-03-28T02:08:49.260012+0000","last_unstale":"2024-03-28T02:10:12.384529+0000","last_undegraded":"2024-03-28T02:10:12.384529+0000","last_fullsized":"2024-03-28T02:10:12.384529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:46:31.233692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459991+0000","last_change":"2024-03-28T02:08:48.447659+0000","last_active":"2024-03-28T02:09:43.459991+0000","last_peered":"2024-03-28T02:09:43.459991+0000","last_clean":"2024-03-28T02:09:43.459991+0000","last_became_active":"2024-03-28T02:08:48.447263+0000","last_became_peered":"2024-03-28T02:08:48.447263+0000","last_unstale":"2024-03-28T02:09:43.459991+0000","last_undegraded":"2024-03-28T02:09:43.459991+0000","last_fullsized":"2024-03-28T02:09:43.459991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:16:32.453528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460498+0000","last_change":"2024-03-28T02:07:33.021769+0000","last_active":"2024-03-28T02:09:43.460498+0000","last_peered":"2024-03-28T02:09:43.460498+0000","last_clean":"2024-03-28T02:09:43.460498+0000","last_became_active":"2024-03-28T02:07:33.021338+0000","last_became_peered":"2024-03-28T02:07:33.021338+0000","last_unstale":"2024-03-28T02:09:43.460498+0000","last_undegraded":"2024-03-28T02:09:43.460498+0000","last_fullsized":"2024-03-28T02:09:43.460498+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:04:07.152098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:04:07.152098+0000","last_clean_scrub_stamp":"2024-03-28T02:04:07.152098+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:01:39.981299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461337+0000","last_change":"2024-03-28T02:08:53.631687+0000","last_active":"2024-03-28T02:09:43.461337+0000","last_peered":"2024-03-28T02:09:43.461337+0000","last_clean":"2024-03-28T02:09:43.461337+0000","last_became_active":"2024-03-28T02:08:53.631291+0000","last_became_peered":"2024-03-28T02:08:53.631291+0000","last_unstale":"2024-03-28T02:09:43.461337+0000","last_undegraded":"2024-03-28T02:09:43.461337+0000","last_fullsized":"2024-03-28T02:09:43.461337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:18:51.563667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.387552+0000","last_change":"2024-03-28T02:08:49.255310+0000","last_active":"2024-03-28T02:10:12.387552+0000","last_peered":"2024-03-28T02:10:12.387552+0000","last_clean":"2024-03-28T02:10:12.387552+0000","last_became_active":"2024-03-28T02:08:49.254923+0000","last_became_peered":"2024-03-28T02:08:49.254923+0000","last_unstale":"2024-03-28T02:10:12.387552+0000","last_undegraded":"2024-03-28T02:10:12.387552+0000","last_fullsized":"2024-03-28T02:10:12.387552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:34.660601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460521+0000","last_change":"2024-03-28T02:08:48.457646+0000","last_active":"2024-03-28T02:09:43.460521+0000","last_peered":"2024-03-28T02:09:43.460521+0000","last_clean":"2024-03-28T02:09:43.460521+0000","last_became_active":"2024-03-28T02:08:48.457344+0000","last_became_peered":"2024-03-28T02:08:48.457344+0000","last_unstale":"2024-03-28T02:09:43.460521+0000","last_undegraded":"2024-03-28T02:09:43.460521+0000","last_fullsized":"2024-03-28T02:09:43.460521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:25:19.077816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460105+0000","last_change":"2024-03-28T02:08:53.486320+0000","last_active":"2024-03-28T02:09:43.460105+0000","last_peered":"2024-03-28T02:09:43.460105+0000","last_clean":"2024-03-28T02:09:43.460105+0000","last_became_active":"2024-03-28T02:08:53.485771+0000","last_became_peered":"2024-03-28T02:08:53.485771+0000","last_unstale":"2024-03-28T02:09:43.460105+0000","last_undegraded":"2024-03-28T02:09:43.460105+0000","last_fullsized":"2024-03-28T02:09:43.460105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:48.452990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461719+0000","last_change":"2024-03-28T02:08:48.458854+0000","last_active":"2024-03-28T02:09:43.461719+0000","last_peered":"2024-03-28T02:09:43.461719+0000","last_clean":"2024-03-28T02:09:43.461719+0000","last_became_active":"2024-03-28T02:08:48.456667+0000","last_became_peered":"2024-03-28T02:08:48.456667+0000","last_unstale":"2024-03-28T02:09:43.461719+0000","last_undegraded":"2024-03-28T02:09:43.461719+0000","last_fullsized":"2024-03-28T02:09:43.461719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:37:54.113797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.391088+0000","last_change":"2024-03-28T02:08:49.255300+0000","last_active":"2024-03-28T02:10:12.391088+0000","last_peered":"2024-03-28T02:10:12.391088+0000","last_clean":"2024-03-28T02:10:12.391088+0000","last_became_active":"2024-03-28T02:08:49.254955+0000","last_became_peered":"2024-03-28T02:08:49.254955+0000","last_unstale":"2024-03-28T02:10:12.391088+0000","last_undegraded":"2024-03-28T02:10:12.391088+0000","last_fullsized":"2024-03-28T02:10:12.391088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:29.093770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.465003+0000","last_change":"2024-03-28T02:08:53.631966+0000","last_active":"2024-03-28T02:09:43.465003+0000","last_peered":"2024-03-28T02:09:43.465003+0000","last_clean":"2024-03-28T02:09:43.465003+0000","last_became_active":"2024-03-28T02:08:53.631457+0000","last_became_peered":"2024-03-28T02:08:53.631457+0000","last_unstale":"2024-03-28T02:09:43.465003+0000","last_undegraded":"2024-03-28T02:09:43.465003+0000","last_fullsized":"2024-03-28T02:09:43.465003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:36:46.623567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.458395+0000","last_change":"2024-03-28T02:08:48.452991+0000","last_active":"2024-03-28T02:09:43.458395+0000","last_peered":"2024-03-28T02:09:43.458395+0000","last_clean":"2024-03-28T02:09:43.458395+0000","last_became_active":"2024-03-28T02:08:48.452248+0000","last_became_peered":"2024-03-28T02:08:48.452248+0000","last_unstale":"2024-03-28T02:09:43.458395+0000","last_undegraded":"2024-03-28T02:09:43.458395+0000","last_fullsized":"2024-03-28T02:09:43.458395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:13:40.100006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.389316+0000","last_change":"2024-03-28T02:08:49.259575+0000","last_active":"2024-03-28T02:10:12.389316+0000","last_peered":"2024-03-28T02:10:12.389316+0000","last_clean":"2024-03-28T02:10:12.389316+0000","last_became_active":"2024-03-28T02:08:49.259255+0000","last_became_peered":"2024-03-28T02:08:49.259255+0000","last_unstale":"2024-03-28T02:10:12.389316+0000","last_undegraded":"2024-03-28T02:10:12.389316+0000","last_fullsized":"2024-03-28T02:10:12.389316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:47:17.741691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457433+0000","last_change":"2024-03-28T02:08:53.488304+0000","last_active":"2024-03-28T02:09:43.457433+0000","last_peered":"2024-03-28T02:09:43.457433+0000","last_clean":"2024-03-28T02:09:43.457433+0000","last_became_active":"2024-03-28T02:08:53.487783+0000","last_became_peered":"2024-03-28T02:08:53.487783+0000","last_unstale":"2024-03-28T02:09:43.457433+0000","last_undegraded":"2024-03-28T02:09:43.457433+0000","last_fullsized":"2024-03-28T02:09:43.457433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:54:50.494685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460346+0000","last_change":"2024-03-28T02:08:48.450795+0000","last_active":"2024-03-28T02:09:43.460346+0000","last_peered":"2024-03-28T02:09:43.460346+0000","last_clean":"2024-03-28T02:09:43.460346+0000","last_became_active":"2024-03-28T02:08:48.450449+0000","last_became_peered":"2024-03-28T02:08:48.450449+0000","last_unstale":"2024-03-28T02:09:43.460346+0000","last_undegraded":"2024-03-28T02:09:43.460346+0000","last_fullsized":"2024-03-28T02:09:43.460346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:44.576188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.393462+0000","last_change":"2024-03-28T02:08:49.264220+0000","last_active":"2024-03-28T02:10:12.393462+0000","last_peered":"2024-03-28T02:10:12.393462+0000","last_clean":"2024-03-28T02:10:12.393462+0000","last_became_active":"2024-03-28T02:08:49.263757+0000","last_became_peered":"2024-03-28T02:08:49.263757+0000","last_unstale":"2024-03-28T02:10:12.393462+0000","last_undegraded":"2024-03-28T02:10:12.393462+0000","last_fullsized":"2024-03-28T02:10:12.393462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:20:44.000256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459943+0000","last_change":"2024-03-28T02:08:53.631904+0000","last_active":"2024-03-28T02:09:43.459943+0000","last_peered":"2024-03-28T02:09:43.459943+0000","last_clean":"2024-03-28T02:09:43.459943+0000","last_became_active":"2024-03-28T02:08:53.631263+0000","last_became_peered":"2024-03-28T02:08:53.631263+0000","last_unstale":"2024-03-28T02:09:43.459943+0000","last_undegraded":"2024-03-28T02:09:43.459943+0000","last_fullsized":"2024-03-28T02:09:43.459943+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:30:53.511632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457483+0000","last_change":"2024-03-28T02:08:48.450272+0000","last_active":"2024-03-28T02:09:43.457483+0000","last_peered":"2024-03-28T02:09:43.457483+0000","last_clean":"2024-03-28T02:09:43.457483+0000","last_became_active":"2024-03-28T02:08:48.449905+0000","last_became_peered":"2024-03-28T02:08:48.449905+0000","last_unstale":"2024-03-28T02:09:43.457483+0000","last_undegraded":"2024-03-28T02:09:43.457483+0000","last_fullsized":"2024-03-28T02:09:43.457483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:56:49.493569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.353622+0000","last_change":"2024-03-28T02:08:49.256816+0000","last_active":"2024-03-28T02:10:12.353622+0000","last_peered":"2024-03-28T02:10:12.353622+0000","last_clean":"2024-03-28T02:10:12.353622+0000","last_became_active":"2024-03-28T02:08:49.256369+0000","last_became_peered":"2024-03-28T02:08:49.256369+0000","last_unstale":"2024-03-28T02:10:12.353622+0000","last_undegraded":"2024-03-28T02:10:12.353622+0000","last_fullsized":"2024-03-28T02:10:12.353622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:39.714619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.465990+0000","last_change":"2024-03-28T02:09:43.465595+0000","last_active":"2024-03-28T02:09:43.465990+0000","last_peered":"2024-03-28T02:09:43.465990+0000","last_clean":"2024-03-28T02:09:43.465990+0000","last_became_active":"2024-03-28T02:09:43.465177+0000","last_became_peered":"2024-03-28T02:09:43.465177+0000","last_unstale":"2024-03-28T02:09:43.465990+0000","last_undegraded":"2024-03-28T02:09:43.465990+0000","last_fullsized":"2024-03-28T02:09:43.465990+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:25:07.219000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.464143+0000","last_change":"2024-03-28T02:08:48.454998+0000","last_active":"2024-03-28T02:09:43.464143+0000","last_peered":"2024-03-28T02:09:43.464143+0000","last_clean":"2024-03-28T02:09:43.464143+0000","last_became_active":"2024-03-28T02:08:48.454674+0000","last_became_peered":"2024-03-28T02:08:48.454674+0000","last_unstale":"2024-03-28T02:09:43.464143+0000","last_undegraded":"2024-03-28T02:09:43.464143+0000","last_fullsized":"2024-03-28T02:09:43.464143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:01:23.432625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.384467+0000","last_change":"2024-03-28T02:08:49.263714+0000","last_active":"2024-03-28T02:10:12.384467+0000","last_peered":"2024-03-28T02:10:12.384467+0000","last_clean":"2024-03-28T02:10:12.384467+0000","last_became_active":"2024-03-28T02:08:49.263335+0000","last_became_peered":"2024-03-28T02:08:49.263335+0000","last_unstale":"2024-03-28T02:10:12.384467+0000","last_undegraded":"2024-03-28T02:10:12.384467+0000","last_fullsized":"2024-03-28T02:10:12.384467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:28:13.153319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459831+0000","last_change":"2024-03-28T02:08:53.496751+0000","last_active":"2024-03-28T02:09:43.459831+0000","last_peered":"2024-03-28T02:09:43.459831+0000","last_clean":"2024-03-28T02:09:43.459831+0000","last_became_active":"2024-03-28T02:08:53.496393+0000","last_became_peered":"2024-03-28T02:08:53.496393+0000","last_unstale":"2024-03-28T02:09:43.459831+0000","last_undegraded":"2024-03-28T02:09:43.459831+0000","last_fullsized":"2024-03-28T02:09:43.459831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:44:13.268278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457433+0000","last_change":"2024-03-28T02:08:48.456026+0000","last_active":"2024-03-28T02:09:43.457433+0000","last_peered":"2024-03-28T02:09:43.457433+0000","last_clean":"2024-03-28T02:09:43.457433+0000","last_became_active":"2024-03-28T02:08:48.455264+0000","last_became_peered":"2024-03-28T02:08:48.455264+0000","last_unstale":"2024-03-28T02:09:43.457433+0000","last_undegraded":"2024-03-28T02:09:43.457433+0000","last_fullsized":"2024-03-28T02:09:43.457433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:21:51.682413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.386231+0000","last_change":"2024-03-28T02:08:49.260416+0000","last_active":"2024-03-28T02:10:12.386231+0000","last_peered":"2024-03-28T02:10:12.386231+0000","last_clean":"2024-03-28T02:10:12.386231+0000","last_became_active":"2024-03-28T02:08:49.260071+0000","last_became_peered":"2024-03-28T02:08:49.260071+0000","last_unstale":"2024-03-28T02:10:12.386231+0000","last_undegraded":"2024-03-28T02:10:12.386231+0000","last_fullsized":"2024-03-28T02:10:12.386231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:34:56.414305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.459713+0000","last_change":"2024-03-28T02:08:53.631633+0000","last_active":"2024-03-28T02:09:43.459713+0000","last_peered":"2024-03-28T02:09:43.459713+0000","last_clean":"2024-03-28T02:09:43.459713+0000","last_became_active":"2024-03-28T02:08:53.631158+0000","last_became_peered":"2024-03-28T02:08:53.631158+0000","last_unstale":"2024-03-28T02:09:43.459713+0000","last_undegraded":"2024-03-28T02:09:43.459713+0000","last_fullsized":"2024-03-28T02:09:43.459713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:37:43.749860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.471538+0000","last_change":"2024-03-28T02:09:43.471538+0000","last_active":"2024-03-28T02:09:43.471538+0000","last_peered":"2024-03-28T02:09:43.471538+0000","last_clean":"2024-03-28T02:09:43.471538+0000","last_became_active":"2024-03-28T02:09:43.466626+0000","last_became_peered":"2024-03-28T02:09:43.466626+0000","last_unstale":"2024-03-28T02:09:43.471538+0000","last_undegraded":"2024-03-28T02:09:43.471538+0000","last_fullsized":"2024-03-28T02:09:43.471538+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:07:15.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402156+0000","last_change":"2024-03-28T02:08:49.268529+0000","last_active":"2024-03-28T02:10:12.402156+0000","last_peered":"2024-03-28T02:10:12.402156+0000","last_clean":"2024-03-28T02:10:12.402156+0000","last_became_active":"2024-03-28T02:08:49.267869+0000","last_became_peered":"2024-03-28T02:08:49.267869+0000","last_unstale":"2024-03-28T02:10:12.402156+0000","last_undegraded":"2024-03-28T02:10:12.402156+0000","last_fullsized":"2024-03-28T02:10:12.402156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:52:34.633616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.457531+0000","last_change":"2024-03-28T02:08:53.495894+0000","last_active":"2024-03-28T02:09:43.457531+0000","last_peered":"2024-03-28T02:09:43.457531+0000","last_clean":"2024-03-28T02:09:43.457531+0000","last_became_active":"2024-03-28T02:08:53.495442+0000","last_became_peered":"2024-03-28T02:08:53.495442+0000","last_unstale":"2024-03-28T02:09:43.457531+0000","last_undegraded":"2024-03-28T02:09:43.457531+0000","last_fullsized":"2024-03-28T02:09:43.457531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:32:33.318061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:24.389475+0000","last_change":"2024-03-28T02:08:48.450376+0000","last_active":"2024-03-28T02:10:24.389475+0000","last_peered":"2024-03-28T02:10:24.389475+0000","last_clean":"2024-03-28T02:10:24.389475+0000","last_became_active":"2024-03-28T02:08:48.449978+0000","last_became_peered":"2024-03-28T02:08:48.449978+0000","last_unstale":"2024-03-28T02:10:24.389475+0000","last_undegraded":"2024-03-28T02:10:24.389475+0000","last_fullsized":"2024-03-28T02:10:24.389475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:33:56.284025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.402305+0000","last_change":"2024-03-28T02:08:49.269011+0000","last_active":"2024-03-28T02:10:12.402305+0000","last_peered":"2024-03-28T02:10:12.402305+0000","last_clean":"2024-03-28T02:10:12.402305+0000","last_became_active":"2024-03-28T02:08:49.268141+0000","last_became_peered":"2024-03-28T02:08:49.268141+0000","last_unstale":"2024-03-28T02:10:12.402305+0000","last_undegraded":"2024-03-28T02:10:12.402305+0000","last_fullsized":"2024-03-28T02:10:12.402305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:54:32.708445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.463419+0000","last_change":"2024-03-28T02:08:53.501549+0000","last_active":"2024-03-28T02:09:43.463419+0000","last_peered":"2024-03-28T02:09:43.463419+0000","last_clean":"2024-03-28T02:09:43.463419+0000","last_became_active":"2024-03-28T02:08:53.501017+0000","last_became_peered":"2024-03-28T02:08:53.501017+0000","last_unstale":"2024-03-28T02:09:43.463419+0000","last_undegraded":"2024-03-28T02:09:43.463419+0000","last_fullsized":"2024-03-28T02:09:43.463419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:33:50.346117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460819+0000","last_change":"2024-03-28T02:08:48.454671+0000","last_active":"2024-03-28T02:09:43.460819+0000","last_peered":"2024-03-28T02:09:43.460819+0000","last_clean":"2024-03-28T02:09:43.460819+0000","last_became_active":"2024-03-28T02:08:48.454329+0000","last_became_peered":"2024-03-28T02:08:48.454329+0000","last_unstale":"2024-03-28T02:09:43.460819+0000","last_undegraded":"2024-03-28T02:09:43.460819+0000","last_fullsized":"2024-03-28T02:09:43.460819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:41:24.087559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388318+0000","last_change":"2024-03-28T02:08:49.259510+0000","last_active":"2024-03-28T02:10:12.388318+0000","last_peered":"2024-03-28T02:10:12.388318+0000","last_clean":"2024-03-28T02:10:12.388318+0000","last_became_active":"2024-03-28T02:08:49.259141+0000","last_became_peered":"2024-03-28T02:08:49.259141+0000","last_unstale":"2024-03-28T02:10:12.388318+0000","last_undegraded":"2024-03-28T02:10:12.388318+0000","last_fullsized":"2024-03-28T02:10:12.388318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:56:50.754576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.461174+0000","last_change":"2024-03-28T02:08:53.488792+0000","last_active":"2024-03-28T02:09:43.461174+0000","last_peered":"2024-03-28T02:09:43.461174+0000","last_clean":"2024-03-28T02:09:43.461174+0000","last_became_active":"2024-03-28T02:08:53.488013+0000","last_became_peered":"2024-03-28T02:08:53.488013+0000","last_unstale":"2024-03-28T02:09:43.461174+0000","last_undegraded":"2024-03-28T02:09:43.461174+0000","last_fullsized":"2024-03-28T02:09:43.461174+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:52.464137+0000","last_clean_scrub_stamp":"2024-03-28T02:08:52.464137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:39:52.006618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:09:43.460300+0000","last_change":"2024-03-28T02:08:48.454609+0000","last_active":"2024-03-28T02:09:43.460300+0000","last_peered":"2024-03-28T02:09:43.460300+0000","last_clean":"2024-03-28T02:09:43.460300+0000","last_became_active":"2024-03-28T02:08:48.454297+0000","last_became_peered":"2024-03-28T02:08:48.454297+0000","last_unstale":"2024-03-28T02:09:43.460300+0000","last_undegraded":"2024-03-28T02:09:43.460300+0000","last_fullsized":"2024-03-28T02:09:43.460300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:47.421314+0000","last_clean_scrub_stamp":"2024-03-28T02:08:47.421314+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:31:42.819338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.379569+0000","last_change":"2024-03-28T02:08:49.263700+0000","last_active":"2024-03-28T02:10:12.379569+0000","last_peered":"2024-03-28T02:10:12.379569+0000","last_clean":"2024-03-28T02:10:12.379569+0000","last_became_active":"2024-03-28T02:08:49.263073+0000","last_became_peered":"2024-03-28T02:08:49.263073+0000","last_unstale":"2024-03-28T02:10:12.379569+0000","last_undegraded":"2024-03-28T02:10:12.379569+0000","last_fullsized":"2024-03-28T02:10:12.379569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:34:19.158284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.389095+0000","last_change":"2024-03-28T02:08:49.265572+0000","last_active":"2024-03-28T02:10:12.389095+0000","last_peered":"2024-03-28T02:10:12.389095+0000","last_clean":"2024-03-28T02:10:12.389095+0000","last_became_active":"2024-03-28T02:08:49.265202+0000","last_became_peered":"2024-03-28T02:08:49.265202+0000","last_unstale":"2024-03-28T02:10:12.389095+0000","last_undegraded":"2024-03-28T02:10:12.389095+0000","last_fullsized":"2024-03-28T02:10:12.389095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:00:05.296762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.396692+0000","last_change":"2024-03-28T02:08:49.266529+0000","last_active":"2024-03-28T02:10:12.396692+0000","last_peered":"2024-03-28T02:10:12.396692+0000","last_clean":"2024-03-28T02:10:12.396692+0000","last_became_active":"2024-03-28T02:08:49.266162+0000","last_became_peered":"2024-03-28T02:08:49.266162+0000","last_unstale":"2024-03-28T02:10:12.396692+0000","last_undegraded":"2024-03-28T02:10:12.396692+0000","last_fullsized":"2024-03-28T02:10:12.396692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:31:40.117845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.392576+0000","last_change":"2024-03-28T02:08:49.264712+0000","last_active":"2024-03-28T02:10:12.392576+0000","last_peered":"2024-03-28T02:10:12.392576+0000","last_clean":"2024-03-28T02:10:12.392576+0000","last_became_active":"2024-03-28T02:08:49.264363+0000","last_became_peered":"2024-03-28T02:08:49.264363+0000","last_unstale":"2024-03-28T02:10:12.392576+0000","last_undegraded":"2024-03-28T02:10:12.392576+0000","last_fullsized":"2024-03-28T02:10:12.392576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:14:09.178725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.395559+0000","last_change":"2024-03-28T02:08:49.261713+0000","last_active":"2024-03-28T02:10:12.395559+0000","last_peered":"2024-03-28T02:10:12.395559+0000","last_clean":"2024-03-28T02:10:12.395559+0000","last_became_active":"2024-03-28T02:08:49.261311+0000","last_became_peered":"2024-03-28T02:08:49.261311+0000","last_unstale":"2024-03-28T02:10:12.395559+0000","last_undegraded":"2024-03-28T02:10:12.395559+0000","last_fullsized":"2024-03-28T02:10:12.395559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:56:54.801575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.383712+0000","last_change":"2024-03-28T02:08:49.256674+0000","last_active":"2024-03-28T02:10:12.383712+0000","last_peered":"2024-03-28T02:10:12.383712+0000","last_clean":"2024-03-28T02:10:12.383712+0000","last_became_active":"2024-03-28T02:08:49.256360+0000","last_became_peered":"2024-03-28T02:08:49.256360+0000","last_unstale":"2024-03-28T02:10:12.383712+0000","last_undegraded":"2024-03-28T02:10:12.383712+0000","last_fullsized":"2024-03-28T02:10:12.383712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:44:14.817569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388927+0000","last_change":"2024-03-28T02:08:49.259829+0000","last_active":"2024-03-28T02:10:12.388927+0000","last_peered":"2024-03-28T02:10:12.388927+0000","last_clean":"2024-03-28T02:10:12.388927+0000","last_became_active":"2024-03-28T02:08:49.259419+0000","last_became_peered":"2024-03-28T02:08:49.259419+0000","last_unstale":"2024-03-28T02:10:12.388927+0000","last_undegraded":"2024-03-28T02:10:12.388927+0000","last_fullsized":"2024-03-28T02:10:12.388927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:49:46.023075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.398783+0000","last_change":"2024-03-28T02:08:49.263116+0000","last_active":"2024-03-28T02:10:12.398783+0000","last_peered":"2024-03-28T02:10:12.398783+0000","last_clean":"2024-03-28T02:10:12.398783+0000","last_became_active":"2024-03-28T02:08:49.262568+0000","last_became_peered":"2024-03-28T02:08:49.262568+0000","last_unstale":"2024-03-28T02:10:12.398783+0000","last_undegraded":"2024-03-28T02:10:12.398783+0000","last_fullsized":"2024-03-28T02:10:12.398783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:29:50.397737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.393017+0000","last_change":"2024-03-28T02:08:49.261449+0000","last_active":"2024-03-28T02:10:12.393017+0000","last_peered":"2024-03-28T02:10:12.393017+0000","last_clean":"2024-03-28T02:10:12.393017+0000","last_became_active":"2024-03-28T02:08:49.261092+0000","last_became_peered":"2024-03-28T02:08:49.261092+0000","last_unstale":"2024-03-28T02:10:12.393017+0000","last_undegraded":"2024-03-28T02:10:12.393017+0000","last_fullsized":"2024-03-28T02:10:12.393017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:51:39.388852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.387857+0000","last_change":"2024-03-28T02:08:49.260711+0000","last_active":"2024-03-28T02:10:12.387857+0000","last_peered":"2024-03-28T02:10:12.387857+0000","last_clean":"2024-03-28T02:10:12.387857+0000","last_became_active":"2024-03-28T02:08:49.259358+0000","last_became_peered":"2024-03-28T02:08:49.259358+0000","last_unstale":"2024-03-28T02:10:12.387857+0000","last_undegraded":"2024-03-28T02:10:12.387857+0000","last_fullsized":"2024-03-28T02:10:12.387857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:27:09.084143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.390756+0000","last_change":"2024-03-28T02:08:49.267145+0000","last_active":"2024-03-28T02:10:12.390756+0000","last_peered":"2024-03-28T02:10:12.390756+0000","last_clean":"2024-03-28T02:10:12.390756+0000","last_became_active":"2024-03-28T02:08:49.266826+0000","last_became_peered":"2024-03-28T02:08:49.266826+0000","last_unstale":"2024-03-28T02:10:12.390756+0000","last_undegraded":"2024-03-28T02:10:12.390756+0000","last_fullsized":"2024-03-28T02:10:12.390756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:37:20.143596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.403822+0000","last_change":"2024-03-28T02:08:49.266752+0000","last_active":"2024-03-28T02:10:12.403822+0000","last_peered":"2024-03-28T02:10:12.403822+0000","last_clean":"2024-03-28T02:10:12.403822+0000","last_became_active":"2024-03-28T02:08:49.266436+0000","last_became_peered":"2024-03-28T02:08:49.266436+0000","last_unstale":"2024-03-28T02:10:12.403822+0000","last_undegraded":"2024-03-28T02:10:12.403822+0000","last_fullsized":"2024-03-28T02:10:12.403822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:52:26.578174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.388673+0000","last_change":"2024-03-28T02:08:49.258934+0000","last_active":"2024-03-28T02:10:12.388673+0000","last_peered":"2024-03-28T02:10:12.388673+0000","last_clean":"2024-03-28T02:10:12.388673+0000","last_became_active":"2024-03-28T02:08:49.258403+0000","last_became_peered":"2024-03-28T02:08:49.258403+0000","last_unstale":"2024-03-28T02:10:12.388673+0000","last_undegraded":"2024-03-28T02:10:12.388673+0000","last_fullsized":"2024-03-28T02:10:12.388673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:44:45.886136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T02:10:12.391565+0000","last_change":"2024-03-28T02:08:49.268456+0000","last_active":"2024-03-28T02:10:12.391565+0000","last_peered":"2024-03-28T02:10:12.391565+0000","last_clean":"2024-03-28T02:10:12.391565+0000","last_became_active":"2024-03-28T02:08:49.267346+0000","last_became_peered":"2024-03-28T02:08:49.267346+0000","last_unstale":"2024-03-28T02:10:12.391565+0000","last_undegraded":"2024-03-28T02:10:12.391565+0000","last_fullsized":"2024-03-28T02:10:12.391565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T02:08:48.430015+0000","last_clean_scrub_stamp":"2024-03-28T02:08:48.430015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:16:47.157938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283904,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73947,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":791,"num_read_kb":472,"num_write":246,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":180,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":221841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":501550,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1159807,"data_compressed":56741,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":501300,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":501566,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1098316,"data_compressed":56741,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496804,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":564827,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":498260,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":564979,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":496512,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":501410,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1085440,"data_stored":1093366,"data_compressed":56741,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36103,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T02:10:28.490 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T02:10:28.490 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T02:10:28.491 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T02:10:28.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T02:10:28.731 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:29.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:29 smithi106 ceph-mon[31965]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:10:29.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:29 smithi106 ceph-mon[31965]: pgmap v344: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:10:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:29 smithi178 ceph-mon[36521]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:10:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:29 smithi178 ceph-mon[36521]: pgmap v344: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:10:29.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:29 smithi153 ceph-mon[36228]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T02:10:29.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:29 smithi153 ceph-mon[36228]: pgmap v344: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:10:30.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.714+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1720841672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdd81021e0 msgr2=0x7fbdd8102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.714+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1720841672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdd81021e0 0x7fbdd8102660 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0099c0 tx=0x7fbdcc009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.714+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1720841672 shutdown_connections 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.714+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1720841672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdd8102ba0 0x7fbdd810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.714+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1720841672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd81081c0 0x7fbdd81085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.714+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1720841672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdd81021e0 0x7fbdd8102660 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.714+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1720841672 >> 172.21.15.106:0/1720841672 conn(0x7fbdd80792c0 msgr2=0x7fbdd80796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.714+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1720841672 shutdown_connections 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.714+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1720841672 wait complete. 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbddcd7f700 1 Processor -- start 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbddcd7f700 1 -- start start 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbddcd7f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdd81021e0 0x7fbdd8074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbddcd7f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd8102ba0 0x7fbdd80749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:30.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbddcd7f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdd81081c0 0x7fbdd8074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:30.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbddcd7f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd8112440 con 0x7fbdd81081c0 2024-03-28T02:10:30.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbddcd7f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd81122c0 con 0x7fbdd8102ba0 2024-03-28T02:10:30.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbddcd7f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd81125c0 con 0x7fbdd81021e0 2024-03-28T02:10:30.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbdd659c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdd81021e0 0x7fbdd8074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:30.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbdd6d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdd81081c0 0x7fbdd8074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:30.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbdd5d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd8102ba0 0x7fbdd80749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:30.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbdd659c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdd81021e0 0x7fbdd8074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:39932/0 (socket says 172.21.15.106:39932) 2024-03-28T02:10:30.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbdd6d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdd81081c0 0x7fbdd8074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55840/0 (socket says 172.21.15.106:55840) 2024-03-28T02:10:30.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.715+0000 7fbdd659c700 1 -- 172.21.15.106:0/1088750696 learned_addr learned my addr 172.21.15.106:0/1088750696 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:30.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdd659c700 1 -- 172.21.15.106:0/1088750696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd8102ba0 msgr2=0x7fbdd80749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:30.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdd659c700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd8102ba0 0x7fbdd80749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdd659c700 1 -- 172.21.15.106:0/1088750696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdd81081c0 msgr2=0x7fbdd8074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:30.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdd659c700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdd81081c0 0x7fbdd8074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdd659c700 1 -- 172.21.15.106:0/1088750696 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd81a6e30 con 0x7fbdd81021e0 2024-03-28T02:10:30.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdd5d9b700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd8102ba0 0x7fbdd80749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:30.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdd659c700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdd81021e0 0x7fbdd8074490 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000ef10 tx=0x7fbdd000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdc77fe700 1 -- 172.21.15.106:0/1088750696 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd000c9c0 con 0x7fbdd81021e0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd81a70c0 con 0x7fbdd81021e0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdc77fe700 1 -- 172.21.15.106:0/1088750696 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbdd0019070 con 0x7fbdd81021e0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbdc77fe700 1 -- 172.21.15.106:0/1088750696 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0007e60 con 0x7fbdd81021e0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.716+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd81a7620 con 0x7fbdd81021e0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.717+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd8068e50 con 0x7fbdd81021e0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.718+0000 7fbdc77fe700 1 -- 172.21.15.106:0/1088750696 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbdd0012050 con 0x7fbdd81021e0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.719+0000 7fbdc77fe700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbdc00a2890 0x7fbdc00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.719+0000 7fbdc77fe700 1 -- 172.21.15.106:0/1088750696 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fbdd00cecc0 con 0x7fbdd81021e0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.719+0000 7fbdd5d9b700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbdc00a2890 0x7fbdc00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:30.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.719+0000 7fbdd5d9b700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbdc00a2890 0x7fbdc00a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8075ed0 tx=0x7fbdcc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:30.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.726+0000 7fbdc77fe700 1 -- 172.21.15.106:0/1088750696 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbdd0010040 con 0x7fbdd81021e0 2024-03-28T02:10:30.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.970+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbdd802da30 con 0x7fbdd81021e0 2024-03-28T02:10:30.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.970+0000 7fbdc77fe700 1 -- 172.21.15.106:0/1088750696 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbdd009b9c0 con 0x7fbdd81021e0 2024-03-28T02:10:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:30.964 INFO:teuthology.orchestra.run.smithi106.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T02:10:30.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbdc00a2890 msgr2=0x7fbdc00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbdc00a2890 0x7fbdc00a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8075ed0 tx=0x7fbdcc005fb0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdd81021e0 msgr2=0x7fbdd8074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdd81021e0 0x7fbdd8074490 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000ef10 tx=0x7fbdd000c3a0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 shutdown_connections 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbdc00a2890 0x7fbdc00a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdd81081c0 0x7fbdd8074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdd81021e0 0x7fbdd8074490 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 --2- 172.21.15.106:0/1088750696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd8102ba0 0x7fbdd80749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 >> 172.21.15.106:0/1088750696 conn(0x7fbdd80792c0 msgr2=0x7fbdd8100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 shutdown_connections 2024-03-28T02:10:30.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:30.972+0000 7fbddcd7f700 1 -- 172.21.15.106:0/1088750696 wait complete. 2024-03-28T02:10:31.728 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:31 smithi106 ceph-mon[31965]: pgmap v345: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:31.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:31 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1088750696' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T02:10:31.757 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T02:10:31.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:10:31.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 -- 172.21.15.106:0/3660391459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c102140 msgr2=0x7fe27c104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 --2- 172.21.15.106:0/3660391459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c102140 0x7fe27c104530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009900 tx=0x7fe26c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 -- 172.21.15.106:0/3660391459 shutdown_connections 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 --2- 172.21.15.106:0/3660391459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe27c1073a0 0x7fe27c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 --2- 172.21.15.106:0/3660391459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe27c104a70 0x7fe27c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 --2- 172.21.15.106:0/3660391459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c102140 0x7fe27c104530 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 -- 172.21.15.106:0/3660391459 >> 172.21.15.106:0/3660391459 conn(0x7fe27c0fbb30 msgr2=0x7fe27c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 -- 172.21.15.106:0/3660391459 shutdown_connections 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 -- 172.21.15.106:0/3660391459 wait complete. 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 Processor -- start 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 -- start start 2024-03-28T02:10:31.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe27c102140 0x7fe27c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.869+0000 7fe281f13640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c104a70 0x7fe27c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe281f13640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe27c1073a0 0x7fe27c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe281f13640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c10d3c0 con 0x7fe27c104a70 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe281f13640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c10d240 con 0x7fe27c102140 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe281f13640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c10d540 con 0x7fe27c1073a0 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe272ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c104a70 0x7fe27c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe272ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c104a70 0x7fe27c197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55862/0 (socket says 172.21.15.106:55862) 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe272ffd640 1 -- 172.21.15.106:0/3014128438 learned_addr learned my addr 172.21.15.106:0/3014128438 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe27b7fe640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe27c102140 0x7fe27c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe27bfff640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe27c1073a0 0x7fe27c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe272ffd640 1 -- 172.21.15.106:0/3014128438 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe27c1073a0 msgr2=0x7fe27c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe272ffd640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe27c1073a0 0x7fe27c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe272ffd640 1 -- 172.21.15.106:0/3014128438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe27c102140 msgr2=0x7fe27c197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe272ffd640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe27c102140 0x7fe27c197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:31.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe272ffd640 1 -- 172.21.15.106:0/3014128438 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c19f630 con 0x7fe27c104a70 2024-03-28T02:10:31.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe272ffd640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c104a70 0x7fe27c197f50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe26400e9f0 tx=0x7fe26400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:31.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe2797fa640 1 -- 172.21.15.106:0/3014128438 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26400ccc0 con 0x7fe27c104a70 2024-03-28T02:10:31.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27c19f920 con 0x7fe27c104a70 2024-03-28T02:10:31.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe2797fa640 1 -- 172.21.15.106:0/3014128438 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe264005500 con 0x7fe27c104a70 2024-03-28T02:10:31.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe2797fa640 1 -- 172.21.15.106:0/3014128438 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264010690 con 0x7fe27c104a70 2024-03-28T02:10:31.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.870+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27c19fe20 con 0x7fe27c104a70 2024-03-28T02:10:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.871+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe27c0fd620 con 0x7fe27c104a70 2024-03-28T02:10:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.872+0000 7fe2797fa640 1 -- 172.21.15.106:0/3014128438 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe2640046c0 con 0x7fe27c104a70 2024-03-28T02:10:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.873+0000 7fe2797fa640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2680a2800 0x7fe2680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.873+0000 7fe27b7fe640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2680a2800 0x7fe2680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.873+0000 7fe2797fa640 1 -- 172.21.15.106:0/3014128438 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fe2640cf2e0 con 0x7fe27c104a70 2024-03-28T02:10:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.873+0000 7fe27b7fe640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2680a2800 0x7fe2680a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe26c02c240 tx=0x7fe26c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:31.876+0000 7fe2797fa640 1 -- 172.21.15.106:0/3014128438 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe264053740 con 0x7fe27c104a70 2024-03-28T02:10:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:31 smithi178 ceph-mon[36521]: pgmap v345: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:31 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1088750696' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T02:10:31.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:31 smithi153 ceph-mon[36228]: pgmap v345: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:31.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:31 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1088750696' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T02:10:32.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.051+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe27c02d070 con 0x7fe27c104a70 2024-03-28T02:10:32.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.053+0000 7fe2797fa640 1 -- 172.21.15.106:0/3014128438 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24705 (secure 0 0 0) 0x7fe27c02d070 con 0x7fe27c104a70 2024-03-28T02:10:32.046 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:32.046 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":91,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:09:43.454083+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:10:32.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2680a2800 msgr2=0x7fe2680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2680a2800 0x7fe2680a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe26c02c240 tx=0x7fe26c02df90 comp rx=0 tx=0).stop 2024-03-28T02:10:32.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c104a70 msgr2=0x7fe27c197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c104a70 0x7fe27c197f50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe26400e9f0 tx=0x7fe26400eeb0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 shutdown_connections 2024-03-28T02:10:32.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe27c1073a0 0x7fe27c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe27c104a70 0x7fe27c197f50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2680a2800 0x7fe2680a4cc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 --2- 172.21.15.106:0/3014128438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe27c102140 0x7fe27c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 >> 172.21.15.106:0/3014128438 conn(0x7fe27c0fbb30 msgr2=0x7fe27c102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:32.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 shutdown_connections 2024-03-28T02:10:32.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.055+0000 7fe281f13640 1 -- 172.21.15.106:0/3014128438 wait complete. 2024-03-28T02:10:32.059 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T02:08:46.952662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T02:08:47.976273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T02:08:51.703592+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T02:10:32.060 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:10:32.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 -- 172.21.15.106:0/1870096495 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd8102160 msgr2=0x7f7dd81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/1870096495 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd8102160 0x7f7dd81089f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0009ca0 tx=0x7f7dd002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 -- 172.21.15.106:0/1870096495 shutdown_connections 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/1870096495 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd8102160 0x7f7dd81089f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/1870096495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd81017a0 0x7f7dd8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/1870096495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd81005b0 0x7f7dd81009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 -- 172.21.15.106:0/1870096495 >> 172.21.15.106:0/1870096495 conn(0x7f7dd80fbb40 msgr2=0x7f7dd80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 -- 172.21.15.106:0/1870096495 shutdown_connections 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 -- 172.21.15.106:0/1870096495 wait complete. 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 Processor -- start 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.172+0000 7f7ddebf4640 1 -- start start 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddebf4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd81005b0 0x7f7dd8196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddebf4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd81017a0 0x7f7dd8196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddebf4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd8102160 0x7f7dd819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddebf4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd810c230 con 0x7f7dd81005b0 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddebf4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd810c0b0 con 0x7f7dd8102160 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddebf4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd810c3b0 con 0x7f7dd81017a0 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddc990640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd81005b0 0x7f7dd8196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddc990640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd81005b0 0x7f7dd8196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55876/0 (socket says 172.21.15.106:55876) 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddc990640 1 -- 172.21.15.106:0/132720581 learned_addr learned my addr 172.21.15.106:0/132720581 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7dd7fff640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd81017a0 0x7f7dd8196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddd191640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd8102160 0x7f7dd819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddc990640 1 -- 172.21.15.106:0/132720581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd81017a0 msgr2=0x7f7dd8196dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddc990640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd81017a0 0x7f7dd8196dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddc990640 1 -- 172.21.15.106:0/132720581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd8102160 msgr2=0x7f7dd819de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddc990640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd8102160 0x7f7dd819de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddc990640 1 -- 172.21.15.106:0/132720581 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd8069c70 con 0x7f7dd81005b0 2024-03-28T02:10:32.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddc990640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd81005b0 0x7f7dd8196880 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00e9c0 tx=0x7f7dcc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:32.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7dd5ffb640 1 -- 172.21.15.106:0/132720581 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc00cd80 con 0x7f7dd81005b0 2024-03-28T02:10:32.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7dd5ffb640 1 -- 172.21.15.106:0/132720581 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dcc0035f0 con 0x7f7dd81005b0 2024-03-28T02:10:32.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd8069f60 con 0x7f7dd81005b0 2024-03-28T02:10:32.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.173+0000 7f7dd5ffb640 1 -- 172.21.15.106:0/132720581 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc010690 con 0x7f7dd81005b0 2024-03-28T02:10:32.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.174+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd806a460 con 0x7f7dd81005b0 2024-03-28T02:10:32.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.175+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd81078d0 con 0x7f7dd81005b0 2024-03-28T02:10:32.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.176+0000 7f7dd5ffb640 1 -- 172.21.15.106:0/132720581 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7dcc0037a0 con 0x7f7dd81005b0 2024-03-28T02:10:32.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.176+0000 7f7dd5ffb640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7dc80a2800 0x7f7dc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.176+0000 7f7dd7fff640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7dc80a2800 0x7f7dc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.176+0000 7f7dd5ffb640 1 -- 172.21.15.106:0/132720581 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f7dcc0cf100 con 0x7f7dd81005b0 2024-03-28T02:10:32.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.177+0000 7f7dd7fff640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7dc80a2800 0x7f7dc80a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8197d90 tx=0x7f7dc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:32.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.180+0000 7f7dd5ffb640 1 -- 172.21.15.106:0/132720581 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7dcc09be60 con 0x7f7dd81005b0 2024-03-28T02:10:32.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.354+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7dd806a720 con 0x7f7dd81005b0 2024-03-28T02:10:32.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.354+0000 7f7dd5ffb640 1 -- 172.21.15.106:0/132720581 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7dcc09b760 con 0x7f7dd81005b0 2024-03-28T02:10:32.348 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:10:32.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7dc80a2800 msgr2=0x7f7dc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7dc80a2800 0x7f7dc80a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8197d90 tx=0x7f7dc4005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:32.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd81005b0 msgr2=0x7f7dd8196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd81005b0 0x7f7dd8196880 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00e9c0 tx=0x7f7dcc00ee80 comp rx=0 tx=0).stop 2024-03-28T02:10:32.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 shutdown_connections 2024-03-28T02:10:32.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7dc80a2800 0x7f7dc80a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd8102160 0x7f7dd819de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd81017a0 0x7f7dd8196dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 --2- 172.21.15.106:0/132720581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dd81005b0 0x7f7dd8196880 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 >> 172.21.15.106:0/132720581 conn(0x7f7dd80fbb40 msgr2=0x7f7dd8104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:32.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 shutdown_connections 2024-03-28T02:10:32.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.356+0000 7f7ddebf4640 1 -- 172.21.15.106:0/132720581 wait complete. 2024-03-28T02:10:32.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T02:10:32.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.474+0000 7f8530819640 1 -- 172.21.15.106:0/1940172189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1016d0 msgr2=0x7f852c101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.474+0000 7f8530819640 1 --2- 172.21.15.106:0/1940172189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1016d0 0x7f852c101af0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f851c009900 tx=0x7f851c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:32.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.474+0000 7f8530819640 1 -- 172.21.15.106:0/1940172189 shutdown_connections 2024-03-28T02:10:32.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.474+0000 7f8530819640 1 --2- 172.21.15.106:0/1940172189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c103280 0x7f852c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.474+0000 7f8530819640 1 --2- 172.21.15.106:0/1940172189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f852c1028c0 0x7f852c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.474+0000 7f8530819640 1 --2- 172.21.15.106:0/1940172189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1016d0 0x7f852c101af0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.474+0000 7f8530819640 1 -- 172.21.15.106:0/1940172189 >> 172.21.15.106:0/1940172189 conn(0x7f852c0fcc40 msgr2=0x7f852c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:32.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.474+0000 7f8530819640 1 -- 172.21.15.106:0/1940172189 shutdown_connections 2024-03-28T02:10:32.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.474+0000 7f8530819640 1 -- 172.21.15.106:0/1940172189 wait complete. 2024-03-28T02:10:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8530819640 1 Processor -- start 2024-03-28T02:10:32.471 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8530819640 1 -- start start 2024-03-28T02:10:32.471 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8530819640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c1016d0 0x7f852c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.471 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8530819640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1028c0 0x7f852c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.471 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8530819640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f852c103280 0x7f852c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.471 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8530819640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c10d390 con 0x7f852c1028c0 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8530819640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c10d210 con 0x7f852c1016d0 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8530819640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c10d510 con 0x7f852c103280 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8521d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1028c0 0x7f852c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f852ad9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f852c103280 0x7f852c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8521d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1028c0 0x7f852c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55902/0 (socket says 172.21.15.106:55902) 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f8521d9b640 1 -- 172.21.15.106:0/3685880677 learned_addr learned my addr 172.21.15.106:0/3685880677 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.475+0000 7f852a59c640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c1016d0 0x7f852c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8521d9b640 1 -- 172.21.15.106:0/3685880677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f852c103280 msgr2=0x7f852c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8521d9b640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f852c103280 0x7f852c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8521d9b640 1 -- 172.21.15.106:0/3685880677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c1016d0 msgr2=0x7f852c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8521d9b640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c1016d0 0x7f852c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8521d9b640 1 -- 172.21.15.106:0/3685880677 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c19f650 con 0x7f852c1028c0 2024-03-28T02:10:32.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f852a59c640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c1016d0 0x7f852c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:32.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8521d9b640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1028c0 0x7f852c197f70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f851400e9f0 tx=0x7f851400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:32.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8523fff640 1 -- 172.21.15.106:0/3685880677 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851400cd80 con 0x7f852c1028c0 2024-03-28T02:10:32.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8523fff640 1 -- 172.21.15.106:0/3685880677 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8514005500 con 0x7f852c1028c0 2024-03-28T02:10:32.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8523fff640 1 -- 172.21.15.106:0/3685880677 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8514010690 con 0x7f852c1028c0 2024-03-28T02:10:32.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852c19f940 con 0x7f852c1028c0 2024-03-28T02:10:32.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.476+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852c19fe40 con 0x7f852c1028c0 2024-03-28T02:10:32.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.477+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852c108a10 con 0x7f852c1028c0 2024-03-28T02:10:32.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.478+0000 7f8523fff640 1 -- 172.21.15.106:0/3685880677 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f85140046c0 con 0x7f852c1028c0 2024-03-28T02:10:32.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.479+0000 7f8523fff640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f85180a2800 0x7f85180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.479+0000 7f852a59c640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f85180a2800 0x7f85180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.479+0000 7f8523fff640 1 -- 172.21.15.106:0/3685880677 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f85140cf190 con 0x7f852c1028c0 2024-03-28T02:10:32.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.479+0000 7f852a59c640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f85180a2800 0x7f85180a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f851c02c170 tx=0x7f851c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:32.476 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.483+0000 7f8523fff640 1 -- 172.21.15.106:0/3685880677 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f851409bef0 con 0x7f852c1028c0 2024-03-28T02:10:32.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.660+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f852c0622a0 con 0x7f852c1028c0 2024-03-28T02:10:32.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.660+0000 7f8523fff640 1 -- 172.21.15.106:0/3685880677 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f851409b7f0 con 0x7f852c1028c0 2024-03-28T02:10:32.653 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:10:32.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f85180a2800 msgr2=0x7f85180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f85180a2800 0x7f85180a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f851c02c170 tx=0x7f851c02df90 comp rx=0 tx=0).stop 2024-03-28T02:10:32.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1028c0 msgr2=0x7f852c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1028c0 0x7f852c197f70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f851400e9f0 tx=0x7f851400edf0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 shutdown_connections 2024-03-28T02:10:32.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f852c103280 0x7f852c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f852c1028c0 0x7f852c197f70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f85180a2800 0x7f85180a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 --2- 172.21.15.106:0/3685880677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c1016d0 0x7f852c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 >> 172.21.15.106:0/3685880677 conn(0x7f852c0fcc40 msgr2=0x7f852c105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:32.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 shutdown_connections 2024-03-28T02:10:32.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.662+0000 7f8530819640 1 -- 172.21.15.106:0/3685880677 wait complete. 2024-03-28T02:10:32.665 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:32 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3014128438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:32.665 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:32 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/132720581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:10:32.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T02:10:32.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/2226828464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d8102160 msgr2=0x7fb0d81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/2226828464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d8102160 0x7fb0d81089f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0009ca0 tx=0x7fb0d002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/2226828464 shutdown_connections 2024-03-28T02:10:32.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/2226828464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d8102160 0x7fb0d81089f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/2226828464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0d81017a0 0x7fb0d8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/2226828464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0d81005b0 0x7fb0d81009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/2226828464 >> 172.21.15.106:0/2226828464 conn(0x7fb0d80fbb40 msgr2=0x7fb0d80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:32.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/2226828464 shutdown_connections 2024-03-28T02:10:32.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/2226828464 wait complete. 2024-03-28T02:10:32.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.778+0000 7fb0ddfe6640 1 Processor -- start 2024-03-28T02:10:32.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0ddfe6640 1 -- start start 2024-03-28T02:10:32.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0ddfe6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0d81005b0 0x7fb0d810af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0ddfe6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d81017a0 0x7fb0d8109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0ddfe6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0d8102160 0x7fb0d8109ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0ddfe6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d810d350 con 0x7fb0d81017a0 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0ddfe6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d810d1d0 con 0x7fb0d8102160 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0ddfe6640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d810d4d0 con 0x7fb0d81005b0 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d81017a0 0x7fb0d8109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d81017a0 0x7fb0d8109570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55918/0 (socket says 172.21.15.106:55918) 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d77fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0d81005b0 0x7fb0d810af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d6ffd640 1 -- 172.21.15.106:0/1507817874 learned_addr learned my addr 172.21.15.106:0/1507817874 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d7fff640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0d8102160 0x7fb0d8109ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d6ffd640 1 -- 172.21.15.106:0/1507817874 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0d81005b0 msgr2=0x7fb0d810af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d6ffd640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0d81005b0 0x7fb0d810af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d6ffd640 1 -- 172.21.15.106:0/1507817874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0d8102160 msgr2=0x7fb0d8109ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d6ffd640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0d8102160 0x7fb0d8109ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d6ffd640 1 -- 172.21.15.106:0/1507817874 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d810a340 con 0x7fb0d81017a0 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d6ffd640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d81017a0 0x7fb0d8109570 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00e9f0 tx=0x7fb0cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d4ff9640 1 -- 172.21.15.106:0/1507817874 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc00cd80 con 0x7fb0d81017a0 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d4ff9640 1 -- 172.21.15.106:0/1507817874 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0cc005500 con 0x7fb0d81017a0 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d81a1ac0 con 0x7fb0d81017a0 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0d4ff9640 1 -- 172.21.15.106:0/1507817874 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc010690 con 0x7fb0d81017a0 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.779+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d81a1fc0 con 0x7fb0d81017a0 2024-03-28T02:10:32.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.780+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d8107920 con 0x7fb0d81017a0 2024-03-28T02:10:32.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.781+0000 7fb0d4ff9640 1 -- 172.21.15.106:0/1507817874 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb0cc0046c0 con 0x7fb0d81017a0 2024-03-28T02:10:32.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.782+0000 7fb0d4ff9640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb0c00a2800 0x7fb0c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:32.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.782+0000 7fb0d77fe640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb0c00a2800 0x7fb0c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:32.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.782+0000 7fb0d4ff9640 1 -- 172.21.15.106:0/1507817874 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fb0cc0cf220 con 0x7fb0d81017a0 2024-03-28T02:10:32.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.782+0000 7fb0d77fe640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb0c00a2800 0x7fb0c00a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8101600 tx=0x7fb0c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:32.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.786+0000 7fb0d4ff9640 1 -- 172.21.15.106:0/1507817874 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb0cc09bf80 con 0x7fb0d81017a0 2024-03-28T02:10:32.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:32 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3014128438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:32.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:32 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/132720581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:10:32.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.960+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0d80622a0 con 0x7fb0d81017a0 2024-03-28T02:10:32.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.960+0000 7fb0d4ff9640 1 -- 172.21.15.106:0/1507817874 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb0cc09b880 con 0x7fb0d81017a0 2024-03-28T02:10:32.953 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 64 2024-03-28T02:10:32.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb0c00a2800 msgr2=0x7fb0c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb0c00a2800 0x7fb0c00a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8101600 tx=0x7fb0c8005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:32.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d81017a0 msgr2=0x7fb0d8109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:32.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d81017a0 0x7fb0d8109570 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00e9f0 tx=0x7fb0cc00edf0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 shutdown_connections 2024-03-28T02:10:32.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb0c00a2800 0x7fb0c00a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0d8102160 0x7fb0d8109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0d81017a0 0x7fb0d8109570 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 --2- 172.21.15.106:0/1507817874 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0d81005b0 0x7fb0d810af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:32.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 >> 172.21.15.106:0/1507817874 conn(0x7fb0d80fbb40 msgr2=0x7fb0d8104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:32.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 shutdown_connections 2024-03-28T02:10:32.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:32.962+0000 7fb0ddfe6640 1 -- 172.21.15.106:0/1507817874 wait complete. 2024-03-28T02:10:32.967 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T02:10:32.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:32 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3014128438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:32.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:32 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/132720581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.077+0000 7f517b6dc640 1 -- 172.21.15.106:0/3066856032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 msgr2=0x7f5174108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.077+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3066856032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 0x7f5174108a00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f516c009ca0 tx=0x7f516c02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.077+0000 7f517b6dc640 1 -- 172.21.15.106:0/3066856032 shutdown_connections 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.077+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3066856032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 0x7f5174108a00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.077+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3066856032 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51741017b0 0x7f5174101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.077+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3066856032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741005c0 0x7f51741009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.077+0000 7f517b6dc640 1 -- 172.21.15.106:0/3066856032 >> 172.21.15.106:0/3066856032 conn(0x7f51740fbb50 msgr2=0x7f51740fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.077+0000 7f517b6dc640 1 -- 172.21.15.106:0/3066856032 shutdown_connections 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.077+0000 7f517b6dc640 1 -- 172.21.15.106:0/3066856032 wait complete. 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f517b6dc640 1 Processor -- start 2024-03-28T02:10:33.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f517b6dc640 1 -- start start 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f517b6dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741005c0 0x7f5174196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f517b6dc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51741017b0 0x7f5174196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f517b6dc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 0x7f517419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f517b6dc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517410c220 con 0x7f5174102170 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f517b6dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517410c0a0 con 0x7f51741005c0 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f517b6dc640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517410c3a0 con 0x7f51741017b0 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5179c79640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 0x7f517419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5179c79640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 0x7f517419dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55942/0 (socket says 172.21.15.106:55942) 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5179c79640 1 -- 172.21.15.106:0/3546589015 learned_addr learned my addr 172.21.15.106:0/3546589015 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5178c77640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51741017b0 0x7f5174196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5179478640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741005c0 0x7f5174196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5179c79640 1 -- 172.21.15.106:0/3546589015 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51741017b0 msgr2=0x7f5174196d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5179c79640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51741017b0 0x7f5174196d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5179c79640 1 -- 172.21.15.106:0/3546589015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741005c0 msgr2=0x7f5174196830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5179c79640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741005c0 0x7f5174196830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5179c79640 1 -- 172.21.15.106:0/3546589015 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517419e450 con 0x7f5174102170 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.078+0000 7f5178c77640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51741017b0 0x7f5174196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.079+0000 7f5179c79640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 0x7f517419dde0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f516c037040 tx=0x7f516c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.079+0000 7f516a7fc640 1 -- 172.21.15.106:0/3546589015 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c041020 con 0x7f5174102170 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.079+0000 7f516a7fc640 1 -- 172.21.15.106:0/3546589015 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f516c034cd0 con 0x7f5174102170 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.079+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517419e6e0 con 0x7f5174102170 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.079+0000 7f516a7fc640 1 -- 172.21.15.106:0/3546589015 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c039040 con 0x7f5174102170 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.079+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517419ebd0 con 0x7f5174102170 2024-03-28T02:10:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.080+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5174107880 con 0x7f5174102170 2024-03-28T02:10:33.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.081+0000 7f516a7fc640 1 -- 172.21.15.106:0/3546589015 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f516c044020 con 0x7f5174102170 2024-03-28T02:10:33.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.081+0000 7f516a7fc640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51600a2800 0x7f51600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.081+0000 7f5179478640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51600a2800 0x7f51600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.081+0000 7f516a7fc640 1 -- 172.21.15.106:0/3546589015 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f516c0f0c40 con 0x7f5174102170 2024-03-28T02:10:33.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.082+0000 7f5179478640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51600a2800 0x7f51600a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f51700095f0 tx=0x7f5170005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:33.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.086+0000 7f516a7fc640 1 -- 172.21.15.106:0/3546589015 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f516c04b020 con 0x7f5174102170 2024-03-28T02:10:33.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.260+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f51740622a0 con 0x7f5174102170 2024-03-28T02:10:33.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.260+0000 7f516a7fc640 1 -- 172.21.15.106:0/3546589015 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f516c0bda30 con 0x7f5174102170 2024-03-28T02:10:33.254 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:10:33.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51600a2800 msgr2=0x7f51600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51600a2800 0x7f51600a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f51700095f0 tx=0x7f5170005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 msgr2=0x7f517419dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 0x7f517419dde0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f516c037040 tx=0x7f516c032f40 comp rx=0 tx=0).stop 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 shutdown_connections 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5174102170 0x7f517419dde0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51741017b0 0x7f5174196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f51600a2800 0x7f51600a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 --2- 172.21.15.106:0/3546589015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741005c0 0x7f5174196830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 >> 172.21.15.106:0/3546589015 conn(0x7f51740fbb50 msgr2=0x7f5174104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 shutdown_connections 2024-03-28T02:10:33.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.263+0000 7f517b6dc640 1 -- 172.21.15.106:0/3546589015 wait complete. 2024-03-28T02:10:33.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:10:33.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.379+0000 7f540112f640 1 -- 172.21.15.106:0/90056910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1017b0 msgr2=0x7f53fc101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.379+0000 7f540112f640 1 --2- 172.21.15.106:0/90056910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1017b0 0x7f53fc101c30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009900 tx=0x7f53f002bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:33.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.379+0000 7f540112f640 1 -- 172.21.15.106:0/90056910 shutdown_connections 2024-03-28T02:10:33.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.379+0000 7f540112f640 1 --2- 172.21.15.106:0/90056910 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53fc102170 0x7f53fc108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.379+0000 7f540112f640 1 --2- 172.21.15.106:0/90056910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1017b0 0x7f53fc101c30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.379+0000 7f540112f640 1 --2- 172.21.15.106:0/90056910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53fc1005c0 0x7f53fc1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.379+0000 7f540112f640 1 -- 172.21.15.106:0/90056910 >> 172.21.15.106:0/90056910 conn(0x7f53fc0fbb30 msgr2=0x7f53fc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:33.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.380+0000 7f540112f640 1 -- 172.21.15.106:0/90056910 shutdown_connections 2024-03-28T02:10:33.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.380+0000 7f540112f640 1 -- 172.21.15.106:0/90056910 wait complete. 2024-03-28T02:10:33.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.380+0000 7f540112f640 1 Processor -- start 2024-03-28T02:10:33.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.380+0000 7f540112f640 1 -- start start 2024-03-28T02:10:33.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f540112f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1005c0 0x7f53fc197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f540112f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53fc1017b0 0x7f53fc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f540112f640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53fc102170 0x7f53fc19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f540112f640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc10c240 con 0x7f53fc1005c0 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f540112f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc10c0c0 con 0x7f53fc1017b0 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f540112f640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc10c3c0 con 0x7f53fc102170 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fad9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1005c0 0x7f53fc197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fad9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1005c0 0x7f53fc197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55958/0 (socket says 172.21.15.106:55958) 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fad9d640 1 -- 172.21.15.106:0/1446480393 learned_addr learned my addr 172.21.15.106:0/1446480393 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fb59e640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53fc102170 0x7f53fc19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fa59c640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53fc1017b0 0x7f53fc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fad9d640 1 -- 172.21.15.106:0/1446480393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53fc102170 msgr2=0x7f53fc19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fad9d640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53fc102170 0x7f53fc19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fad9d640 1 -- 172.21.15.106:0/1446480393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53fc1017b0 msgr2=0x7f53fc197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fad9d640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53fc1017b0 0x7f53fc197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fad9d640 1 -- 172.21.15.106:0/1446480393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc19f680 con 0x7f53fc1005c0 2024-03-28T02:10:33.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53fad9d640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1005c0 0x7f53fc197a60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00e9c0 tx=0x7f53ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:33.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53e3fff640 1 -- 172.21.15.106:0/1446480393 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec00cde0 con 0x7f53fc1005c0 2024-03-28T02:10:33.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53e3fff640 1 -- 172.21.15.106:0/1446480393 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f53ec010910 con 0x7f53fc1005c0 2024-03-28T02:10:33.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53fc19f970 con 0x7f53fc1005c0 2024-03-28T02:10:33.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.381+0000 7f53e3fff640 1 -- 172.21.15.106:0/1446480393 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec018a00 con 0x7f53fc1005c0 2024-03-28T02:10:33.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.382+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53fc19fe90 con 0x7f53fc1005c0 2024-03-28T02:10:33.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.383+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53fc1a00c0 con 0x7f53fc1005c0 2024-03-28T02:10:33.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.383+0000 7f53e3fff640 1 -- 172.21.15.106:0/1446480393 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f53ec01f020 con 0x7f53fc1005c0 2024-03-28T02:10:33.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.384+0000 7f53e3fff640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f53e40a2800 0x7f53e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.384+0000 7f53fa59c640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f53e40a2800 0x7f53e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.384+0000 7f53e3fff640 1 -- 172.21.15.106:0/1446480393 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f53ec0cef80 con 0x7f53fc1005c0 2024-03-28T02:10:33.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.385+0000 7f53fa59c640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f53e40a2800 0x7f53e40a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009230 tx=0x7f53f0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:33.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.389+0000 7f53e3fff640 1 -- 172.21.15.106:0/1446480393 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f53ec09bce0 con 0x7f53fc1005c0 2024-03-28T02:10:33.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.611+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53fc1a0290 con 0x7f53fc1005c0 2024-03-28T02:10:33.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.612+0000 7f53e3fff640 1 -- 172.21.15.106:0/1446480393 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11474 (secure 0 0 0) 0x7f53ec09b5e0 con 0x7f53fc1005c0 2024-03-28T02:10:33.605 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:33.605 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:09:11.288485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24499,"mds_2":24460,"mds_3":24446,"mds_4":24415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14625":{"gid":14625,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14637":{"gid":14637,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24446":{"gid":24446,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24415,24499,24509,24446,24460]},"id":1}]} 2024-03-28T02:10:33.606 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 28 2024-03-28T02:10:33.607 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f53e40a2800 msgr2=0x7f53e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f53e40a2800 0x7f53e40a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009230 tx=0x7f53f0005fd0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1005c0 msgr2=0x7f53fc197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1005c0 0x7f53fc197a60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00e9c0 tx=0x7f53ec00ee80 comp rx=0 tx=0).stop 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 shutdown_connections 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53fc102170 0x7f53fc19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f53e40a2800 0x7f53e40a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53fc1017b0 0x7f53fc197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 --2- 172.21.15.106:0/1446480393 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53fc1005c0 0x7f53fc197a60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 >> 172.21.15.106:0/1446480393 conn(0x7f53fc0fbb30 msgr2=0x7f53fc0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 shutdown_connections 2024-03-28T02:10:33.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.614+0000 7f540112f640 1 -- 172.21.15.106:0/1446480393 wait complete. 2024-03-28T02:10:33.619 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:10:33.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.732+0000 7ff58706e640 1 -- 172.21.15.106:0/394043814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff580104ad0 msgr2=0x7ff580106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.732+0000 7ff58706e640 1 --2- 172.21.15.106:0/394043814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff580104ad0 0x7ff580106ec0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009900 tx=0x7ff56c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:33.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.732+0000 7ff58706e640 1 -- 172.21.15.106:0/394043814 shutdown_connections 2024-03-28T02:10:33.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.732+0000 7ff58706e640 1 --2- 172.21.15.106:0/394043814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff580107400 0x7ff5801097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.732+0000 7ff58706e640 1 --2- 172.21.15.106:0/394043814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff580104ad0 0x7ff580106ec0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.732+0000 7ff58706e640 1 --2- 172.21.15.106:0/394043814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5801021a0 0x7ff580104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.732+0000 7ff58706e640 1 -- 172.21.15.106:0/394043814 >> 172.21.15.106:0/394043814 conn(0x7ff5800fbb30 msgr2=0x7ff5800fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.732+0000 7ff58706e640 1 -- 172.21.15.106:0/394043814 shutdown_connections 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.732+0000 7ff58706e640 1 -- 172.21.15.106:0/394043814 wait complete. 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58706e640 1 Processor -- start 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58706e640 1 -- start start 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58706e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5801021a0 0x7ff580101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58706e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff580104ad0 0x7ff580100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58706e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff580107400 0x7ff580100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58706e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58010d390 con 0x7ff5801021a0 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58706e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58010d210 con 0x7ff580104ad0 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58706e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58010d510 con 0x7ff580107400 2024-03-28T02:10:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff584e0a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5801021a0 0x7ff580101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff584e0a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5801021a0 0x7ff580101b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55970/0 (socket says 172.21.15.106:55970) 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff584e0a640 1 -- 172.21.15.106:0/3448499985 learned_addr learned my addr 172.21.15.106:0/3448499985 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58560b640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff580107400 0x7ff580100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff57ffff640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff580104ad0 0x7ff580100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff584e0a640 1 -- 172.21.15.106:0/3448499985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff580107400 msgr2=0x7ff580100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff584e0a640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff580107400 0x7ff580100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff584e0a640 1 -- 172.21.15.106:0/3448499985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff580104ad0 msgr2=0x7ff580100150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff584e0a640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff580104ad0 0x7ff580100150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff584e0a640 1 -- 172.21.15.106:0/3448499985 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff580100f20 con 0x7ff5801021a0 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff57ffff640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff580104ad0 0x7ff580100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff584e0a640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5801021a0 0x7ff580101b10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff57400e9f0 tx=0x7ff57400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff57dffb640 1 -- 172.21.15.106:0/3448499985 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57400cd80 con 0x7ff5801021a0 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff57dffb640 1 -- 172.21.15.106:0/3448499985 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff574010910 con 0x7ff5801021a0 2024-03-28T02:10:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5801a1b00 con 0x7ff5801021a0 2024-03-28T02:10:33.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.733+0000 7ff57dffb640 1 -- 172.21.15.106:0/3448499985 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574018990 con 0x7ff5801021a0 2024-03-28T02:10:33.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.734+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5801a2020 con 0x7ff5801021a0 2024-03-28T02:10:33.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.735+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5800fd620 con 0x7ff5801021a0 2024-03-28T02:10:33.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.737+0000 7ff57dffb640 1 -- 172.21.15.106:0/3448499985 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff574010ac0 con 0x7ff5801021a0 2024-03-28T02:10:33.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.737+0000 7ff57dffb640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2800 0x7ff5700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:33.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.737+0000 7ff57dffb640 1 -- 172.21.15.106:0/3448499985 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7ff5740cf560 con 0x7ff5801021a0 2024-03-28T02:10:33.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.737+0000 7ff57ffff640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2800 0x7ff5700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:33.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.738+0000 7ff57ffff640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2800 0x7ff5700a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff56c02c170 tx=0x7ff56c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:33.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.740+0000 7ff57dffb640 1 -- 172.21.15.106:0/3448499985 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff57409c2c0 con 0x7ff5801021a0 2024-03-28T02:10:33.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:33 smithi106 ceph-mon[31965]: pgmap v346: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:33 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3685880677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:10:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:33 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1507817874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:10:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:33 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3546589015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:10:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:33 smithi178 ceph-mon[36521]: pgmap v346: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:33 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3685880677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:10:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:33 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1507817874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:10:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:33 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3546589015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:10:33.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.922+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff58002d240 con 0x7ff5801021a0 2024-03-28T02:10:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.924+0000 7ff57dffb640 1 -- 172.21.15.106:0/3448499985 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24705 (secure 0 0 0) 0x7ff57409bbc0 con 0x7ff5801021a0 2024-03-28T02:10:33.917 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:33.917 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":91,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:09:43.454083+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:10:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2800 msgr2=0x7ff5700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2800 0x7ff5700a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff56c02c170 tx=0x7ff56c02df90 comp rx=0 tx=0).stop 2024-03-28T02:10:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5801021a0 msgr2=0x7ff580101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5801021a0 0x7ff580101b10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff57400e9f0 tx=0x7ff57400eeb0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 shutdown_connections 2024-03-28T02:10:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff580107400 0x7ff580100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff5700a2800 0x7ff5700a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff580104ad0 0x7ff580100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 --2- 172.21.15.106:0/3448499985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5801021a0 0x7ff580101b10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 >> 172.21.15.106:0/3448499985 conn(0x7ff5800fbb30 msgr2=0x7ff580102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 shutdown_connections 2024-03-28T02:10:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:33.926+0000 7ff58706e640 1 -- 172.21.15.106:0/3448499985 wait complete. 2024-03-28T02:10:33.931 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T02:08:46.952662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T02:08:47.976273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T02:08:51.703592+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T02:10:33.932 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:10:33.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:33 smithi153 ceph-mon[36228]: pgmap v346: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:33.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:33 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3685880677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:10:33.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:33 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1507817874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:10:33.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:33 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3546589015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:10:34.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.045+0000 7f27e8add640 1 -- 172.21.15.106:0/2881612241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e4102150 msgr2=0x7f27e41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.045+0000 7f27e8add640 1 --2- 172.21.15.106:0/2881612241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e4102150 0x7f27e41089e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009ca0 tx=0x7f27dc02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.045+0000 7f27e8add640 1 -- 172.21.15.106:0/2881612241 shutdown_connections 2024-03-28T02:10:34.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.045+0000 7f27e8add640 1 --2- 172.21.15.106:0/2881612241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e4102150 0x7f27e41089e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.045+0000 7f27e8add640 1 --2- 172.21.15.106:0/2881612241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27e4101790 0x7f27e4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.045+0000 7f27e8add640 1 --2- 172.21.15.106:0/2881612241 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e41005a0 0x7f27e41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.045+0000 7f27e8add640 1 -- 172.21.15.106:0/2881612241 >> 172.21.15.106:0/2881612241 conn(0x7f27e40fbb30 msgr2=0x7f27e40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:34.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.045+0000 7f27e8add640 1 -- 172.21.15.106:0/2881612241 shutdown_connections 2024-03-28T02:10:34.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.045+0000 7f27e8add640 1 -- 172.21.15.106:0/2881612241 wait complete. 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.046+0000 7f27e8add640 1 Processor -- start 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.046+0000 7f27e8add640 1 -- start start 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e8add640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27e41005a0 0x7f27e4195750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e8add640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e4101790 0x7f27e4195c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e8add640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e4102150 0x7f27e419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e8add640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e410b120 con 0x7f27e41005a0 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e8add640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e410afa0 con 0x7f27e4102150 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e8add640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e410b2a0 con 0x7f27e4101790 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27e41005a0 0x7f27e4195750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27e41005a0 0x7f27e4195750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55990/0 (socket says 172.21.15.106:55990) 2024-03-28T02:10:34.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e259c640 1 -- 172.21.15.106:0/1577740931 learned_addr learned my addr 172.21.15.106:0/1577740931 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e2d9d640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e4102150 0x7f27e419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27d9d9b640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e4101790 0x7f27e4195c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e259c640 1 -- 172.21.15.106:0/1577740931 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e4101790 msgr2=0x7f27e4195c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e259c640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e4101790 0x7f27e4195c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e259c640 1 -- 172.21.15.106:0/1577740931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e4102150 msgr2=0x7f27e419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e259c640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e4102150 0x7f27e419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e259c640 1 -- 172.21.15.106:0/1577740931 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e419d370 con 0x7f27e41005a0 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e259c640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27e41005a0 0x7f27e4195750 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f27d400e9c0 tx=0x7f27d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27dbfff640 1 -- 172.21.15.106:0/1577740931 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d400cd80 con 0x7f27e41005a0 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27dbfff640 1 -- 172.21.15.106:0/1577740931 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27d40035f0 con 0x7f27e41005a0 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e419d660 con 0x7f27e41005a0 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27dbfff640 1 -- 172.21.15.106:0/1577740931 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4010690 con 0x7f27e41005a0 2024-03-28T02:10:34.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.047+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e419dbb0 con 0x7f27e41005a0 2024-03-28T02:10:34.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.049+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e41078c0 con 0x7f27e41005a0 2024-03-28T02:10:34.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.050+0000 7f27dbfff640 1 -- 172.21.15.106:0/1577740931 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f27d40037a0 con 0x7f27e41005a0 2024-03-28T02:10:34.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.051+0000 7f27dbfff640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27d00a2800 0x7f27d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.051+0000 7f27d9d9b640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27d00a2800 0x7f27d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.051+0000 7f27dbfff640 1 -- 172.21.15.106:0/1577740931 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f27d40cf150 con 0x7f27e41005a0 2024-03-28T02:10:34.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.051+0000 7f27d9d9b640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27d00a2800 0x7f27d00a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00b800 tx=0x7f27cc00d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:34.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.055+0000 7f27dbfff640 1 -- 172.21.15.106:0/1577740931 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f27d409beb0 con 0x7f27e41005a0 2024-03-28T02:10:34.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.237+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27e40622d0 con 0x7f27e41005a0 2024-03-28T02:10:34.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.237+0000 7f27dbfff640 1 -- 172.21.15.106:0/1577740931 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27d409b7b0 con 0x7f27e41005a0 2024-03-28T02:10:34.230 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.239+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27d00a2800 msgr2=0x7f27d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.239+0000 7f27e8add640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27d00a2800 0x7f27d00a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00b800 tx=0x7f27cc00d040 comp rx=0 tx=0).stop 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.239+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27e41005a0 msgr2=0x7f27e4195750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.239+0000 7f27e8add640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27e41005a0 0x7f27e4195750 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f27d400e9c0 tx=0x7f27d400ee80 comp rx=0 tx=0).stop 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.240+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 shutdown_connections 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.240+0000 7f27e8add640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f27d00a2800 0x7f27d00a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.240+0000 7f27e8add640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e4102150 0x7f27e419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.240+0000 7f27e8add640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e4101790 0x7f27e4195c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.240+0000 7f27e8add640 1 --2- 172.21.15.106:0/1577740931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27e41005a0 0x7f27e4195750 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.240+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 >> 172.21.15.106:0/1577740931 conn(0x7f27e40fbb30 msgr2=0x7f27e4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.240+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 shutdown_connections 2024-03-28T02:10:34.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.240+0000 7f27e8add640 1 -- 172.21.15.106:0/1577740931 wait complete. 2024-03-28T02:10:34.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T02:10:34.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.357+0000 7f4662f76640 1 -- 172.21.15.106:0/690734559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c107380 msgr2=0x7f465c1097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.357+0000 7f4662f76640 1 --2- 172.21.15.106:0/690734559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c107380 0x7f465c1097d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4654009ca0 tx=0x7f465402c0f0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.357+0000 7f4662f76640 1 -- 172.21.15.106:0/690734559 shutdown_connections 2024-03-28T02:10:34.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.357+0000 7f4662f76640 1 --2- 172.21.15.106:0/690734559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c107380 0x7f465c1097d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 --2- 172.21.15.106:0/690734559 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f465c104a50 0x7f465c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 --2- 172.21.15.106:0/690734559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f465c102120 0x7f465c104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 -- 172.21.15.106:0/690734559 >> 172.21.15.106:0/690734559 conn(0x7f465c0fbb30 msgr2=0x7f465c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:34.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 -- 172.21.15.106:0/690734559 shutdown_connections 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 -- 172.21.15.106:0/690734559 wait complete. 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 Processor -- start 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 -- start start 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c102120 0x7f465c101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f465c104a50 0x7f465c1000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f465c107380 0x7f465c100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c10d320 con 0x7f465c102120 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c10d1a0 con 0x7f465c107380 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4662f76640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c10d4a0 con 0x7f465c104a50 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4660d12640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c102120 0x7f465c101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4660d12640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c102120 0x7f465c101a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56000/0 (socket says 172.21.15.106:56000) 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4660d12640 1 -- 172.21.15.106:0/2537402267 learned_addr learned my addr 172.21.15.106:0/2537402267 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f465bfff640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f465c104a50 0x7f465c1000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.358+0000 7f4661513640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f465c107380 0x7f465c100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4660d12640 1 -- 172.21.15.106:0/2537402267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f465c104a50 msgr2=0x7f465c1000d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4660d12640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f465c104a50 0x7f465c1000d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4660d12640 1 -- 172.21.15.106:0/2537402267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f465c107380 msgr2=0x7f465c100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4660d12640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f465c107380 0x7f465c100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4660d12640 1 -- 172.21.15.106:0/2537402267 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c100ea0 con 0x7f465c102120 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f465bfff640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f465c104a50 0x7f465c1000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4660d12640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c102120 0x7f465c101a90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f465000e9f0 tx=0x7f465000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4659ffb640 1 -- 172.21.15.106:0/2537402267 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465000cd80 con 0x7f465c102120 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465c1a1af0 con 0x7f465c102120 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4659ffb640 1 -- 172.21.15.106:0/2537402267 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4650010910 con 0x7f465c102120 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4659ffb640 1 -- 172.21.15.106:0/2537402267 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650018950 con 0x7f465c102120 2024-03-28T02:10:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.359+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465c1a2090 con 0x7f465c102120 2024-03-28T02:10:34.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.360+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465c068490 con 0x7f465c102120 2024-03-28T02:10:34.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.362+0000 7f4659ffb640 1 -- 172.21.15.106:0/2537402267 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4650010ac0 con 0x7f465c102120 2024-03-28T02:10:34.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.362+0000 7f4659ffb640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f464c0a2800 0x7f464c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.363+0000 7f4659ffb640 1 -- 172.21.15.106:0/2537402267 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f46500cf140 con 0x7f465c102120 2024-03-28T02:10:34.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.363+0000 7f465bfff640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f464c0a2800 0x7f464c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.363+0000 7f465bfff640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f464c0a2800 0x7f464c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f465c101800 tx=0x7f4648005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:34.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.367+0000 7f4659ffb640 1 -- 172.21.15.106:0/2537402267 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f465009b850 con 0x7f465c102120 2024-03-28T02:10:34.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.548+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f465c0622a0 con 0x7f465c102120 2024-03-28T02:10:34.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.549+0000 7f4659ffb640 1 -- 172.21.15.106:0/2537402267 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f465009f8f0 con 0x7f465c102120 2024-03-28T02:10:34.542 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:10:34.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f464c0a2800 msgr2=0x7f464c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f464c0a2800 0x7f464c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f465c101800 tx=0x7f4648005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:34.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c102120 msgr2=0x7f465c101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c102120 0x7f465c101a90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f465000e9f0 tx=0x7f465000eeb0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 shutdown_connections 2024-03-28T02:10:34.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f464c0a2800 0x7f464c0a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f465c107380 0x7f465c100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f465c104a50 0x7f465c1000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 --2- 172.21.15.106:0/2537402267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f465c102120 0x7f465c101a90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 >> 172.21.15.106:0/2537402267 conn(0x7f465c0fbb30 msgr2=0x7f465c1029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:34.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 shutdown_connections 2024-03-28T02:10:34.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.551+0000 7f4662f76640 1 -- 172.21.15.106:0/2537402267 wait complete. 2024-03-28T02:10:34.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T02:10:34.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3922555780 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481005b0 msgr2=0x7fd6481009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3922555780 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481005b0 0x7fd6481009d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd638009900 tx=0x7fd63802bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:34.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3922555780 shutdown_connections 2024-03-28T02:10:34.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3922555780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd648102160 0x7fd6481089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3922555780 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6481017a0 0x7fd648101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3922555780 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481005b0 0x7fd6481009d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3922555780 >> 172.21.15.106:0/3922555780 conn(0x7fd6480fbb40 msgr2=0x7fd6480fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:34.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3922555780 shutdown_connections 2024-03-28T02:10:34.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3922555780 wait complete. 2024-03-28T02:10:34.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 Processor -- start 2024-03-28T02:10:34.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.667+0000 7fd64e9c4640 1 -- start start 2024-03-28T02:10:34.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd64e9c4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6481005b0 0x7fd648199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd64e9c4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481017a0 0x7fd64819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd64e9c4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd648102160 0x7fd6481a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd64e9c4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64810c220 con 0x7fd6481017a0 2024-03-28T02:10:34.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd64e9c4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64810c0a0 con 0x7fd648102160 2024-03-28T02:10:34.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd64e9c4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64810c3a0 con 0x7fd6481005b0 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd6477fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481017a0 0x7fd64819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd6477fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481017a0 0x7fd64819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56018/0 (socket says 172.21.15.106:56018) 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd6477fe640 1 -- 172.21.15.106:0/3316455708 learned_addr learned my addr 172.21.15.106:0/3316455708 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd647fff640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6481005b0 0x7fd648199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd64cf61640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd648102160 0x7fd6481a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd6477fe640 1 -- 172.21.15.106:0/3316455708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6481005b0 msgr2=0x7fd648199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd6477fe640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6481005b0 0x7fd648199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd6477fe640 1 -- 172.21.15.106:0/3316455708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd648102160 msgr2=0x7fd6481a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd6477fe640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd648102160 0x7fd6481a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd6477fe640 1 -- 172.21.15.106:0/3316455708 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6481a1880 con 0x7fd6481017a0 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.668+0000 7fd6477fe640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481017a0 0x7fd64819a1b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00e9f0 tx=0x7fd63c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:34.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.669+0000 7fd6457fa640 1 -- 172.21.15.106:0/3316455708 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c00cd80 con 0x7fd6481017a0 2024-03-28T02:10:34.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.669+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6481a1b70 con 0x7fd6481017a0 2024-03-28T02:10:34.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.669+0000 7fd6457fa640 1 -- 172.21.15.106:0/3316455708 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd63c005500 con 0x7fd6481017a0 2024-03-28T02:10:34.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.669+0000 7fd6457fa640 1 -- 172.21.15.106:0/3316455708 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c010690 con 0x7fd6481017a0 2024-03-28T02:10:34.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.669+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6481a2090 con 0x7fd6481017a0 2024-03-28T02:10:34.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.670+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6480688e0 con 0x7fd6481017a0 2024-03-28T02:10:34.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.671+0000 7fd6457fa640 1 -- 172.21.15.106:0/3316455708 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd63c0046c0 con 0x7fd6481017a0 2024-03-28T02:10:34.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.672+0000 7fd6457fa640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd6300a2800 0x7fd6300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.672+0000 7fd647fff640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd6300a2800 0x7fd6300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.672+0000 7fd6457fa640 1 -- 172.21.15.106:0/3316455708 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fd63c0cf450 con 0x7fd6481017a0 2024-03-28T02:10:34.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.672+0000 7fd647fff640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd6300a2800 0x7fd6300a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd638000c00 tx=0x7fd638005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:34.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.675+0000 7fd6457fa640 1 -- 172.21.15.106:0/3316455708 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd63c053740 con 0x7fd6481017a0 2024-03-28T02:10:34.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:34 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1446480393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:10:34.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:34 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3448499985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:34.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:34 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1577740931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:10:34.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:34 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2537402267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:10:34.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.851+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd6481a24e0 con 0x7fd6481017a0 2024-03-28T02:10:34.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.852+0000 7fd6457fa640 1 -- 172.21.15.106:0/3316455708 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd63c09c1b0 con 0x7fd6481017a0 2024-03-28T02:10:34.845 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 64 2024-03-28T02:10:34.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd6300a2800 msgr2=0x7fd6300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd6300a2800 0x7fd6300a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd638000c00 tx=0x7fd638005fd0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481017a0 msgr2=0x7fd64819a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481017a0 0x7fd64819a1b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00e9f0 tx=0x7fd63c00edf0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 shutdown_connections 2024-03-28T02:10:34.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd6300a2800 0x7fd6300a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd648102160 0x7fd6481a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6481017a0 0x7fd64819a1b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 --2- 172.21.15.106:0/3316455708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6481005b0 0x7fd648199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 >> 172.21.15.106:0/3316455708 conn(0x7fd6480fbb40 msgr2=0x7fd6480fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:34.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 shutdown_connections 2024-03-28T02:10:34.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.854+0000 7fd64e9c4640 1 -- 172.21.15.106:0/3316455708 wait complete. 2024-03-28T02:10:34.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T02:10:34.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:34 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1446480393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:10:34.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:34 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3448499985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:34.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:34 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1577740931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:10:34.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:34 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2537402267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:10:34.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 -- 172.21.15.106:0/674477971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c01028a0 msgr2=0x7f62c0102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/674477971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c01028a0 0x7f62c0102d20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f62b4009900 tx=0x7f62b402bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:34.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 -- 172.21.15.106:0/674477971 shutdown_connections 2024-03-28T02:10:34.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/674477971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62c0103260 0x7f62c0109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/674477971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c01028a0 0x7f62c0102d20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/674477971 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f62c01016b0 0x7f62c0101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 -- 172.21.15.106:0/674477971 >> 172.21.15.106:0/674477971 conn(0x7f62c00fcc40 msgr2=0x7f62c00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 -- 172.21.15.106:0/674477971 shutdown_connections 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 -- 172.21.15.106:0/674477971 wait complete. 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.970+0000 7f62c5dc6640 1 Processor -- start 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62c5dc6640 1 -- start start 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62c5dc6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f62c01016b0 0x7f62c01979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62c5dc6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62c01028a0 0x7f62c0197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62c5dc6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c0103260 0x7f62c019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62c5dc6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c010d360 con 0x7f62c0103260 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62c5dc6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c010d1e0 con 0x7f62c01028a0 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62c5dc6640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c010d4e0 con 0x7f62c01016b0 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c0103260 0x7f62c019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c0103260 0x7f62c019ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56040/0 (socket says 172.21.15.106:56040) 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bffff640 1 -- 172.21.15.106:0/3805916451 learned_addr learned my addr 172.21.15.106:0/3805916451 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bf7fe640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f62c01016b0 0x7f62c01979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62beffd640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62c01028a0 0x7f62c0197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bffff640 1 -- 172.21.15.106:0/3805916451 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f62c01016b0 msgr2=0x7f62c01979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bffff640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f62c01016b0 0x7f62c01979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bffff640 1 -- 172.21.15.106:0/3805916451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62c01028a0 msgr2=0x7f62c0197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bffff640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62c01028a0 0x7f62c0197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bffff640 1 -- 172.21.15.106:0/3805916451 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c019f5c0 con 0x7f62c0103260 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bffff640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c0103260 0x7f62c019ef50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f62b800eb10 tx=0x7f62b800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.971+0000 7f62bcff9640 1 -- 172.21.15.106:0/3805916451 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8004b20 con 0x7f62c0103260 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.972+0000 7f62bcff9640 1 -- 172.21.15.106:0/3805916451 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62b8004cc0 con 0x7f62c0103260 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.972+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c019f8b0 con 0x7f62c0103260 2024-03-28T02:10:34.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.972+0000 7f62bcff9640 1 -- 172.21.15.106:0/3805916451 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8010760 con 0x7f62c0103260 2024-03-28T02:10:34.968 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.972+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c019fd80 con 0x7f62c0103260 2024-03-28T02:10:34.968 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.973+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c01089d0 con 0x7f62c0103260 2024-03-28T02:10:34.968 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.973+0000 7f62bcff9640 1 -- 172.21.15.106:0/3805916451 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f62b8005020 con 0x7f62c0103260 2024-03-28T02:10:34.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.974+0000 7f62bcff9640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f62a80a2800 0x7f62a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:34.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.974+0000 7f62bf7fe640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f62a80a2800 0x7f62a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:34.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.974+0000 7f62bcff9640 1 -- 172.21.15.106:0/3805916451 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f62b80cf310 con 0x7f62c0103260 2024-03-28T02:10:34.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.974+0000 7f62bf7fe640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f62a80a2800 0x7f62a80a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f62c0102700 tx=0x7f62b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:34.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:34.978+0000 7f62bcff9640 1 -- 172.21.15.106:0/3805916451 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f62b809c070 con 0x7f62c0103260 2024-03-28T02:10:34.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:34 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1446480393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:10:34.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:34 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3448499985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:34.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:34 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1577740931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:10:34.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:34 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2537402267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:10:35.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.152+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f62c00622a0 con 0x7f62c0103260 2024-03-28T02:10:35.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.152+0000 7f62bcff9640 1 -- 172.21.15.106:0/3805916451 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f62b809b970 con 0x7f62c0103260 2024-03-28T02:10:35.146 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f62a80a2800 msgr2=0x7f62a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f62a80a2800 0x7f62a80a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f62c0102700 tx=0x7f62b0005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c0103260 msgr2=0x7f62c019ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c0103260 0x7f62c019ef50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f62b800eb10 tx=0x7f62b800c2d0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 shutdown_connections 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c0103260 0x7f62c019ef50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f62a80a2800 0x7f62a80a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62c01028a0 0x7f62c0197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 --2- 172.21.15.106:0/3805916451 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f62c01016b0 0x7f62c01979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.154+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 >> 172.21.15.106:0/3805916451 conn(0x7f62c00fcc40 msgr2=0x7f62c0105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:35.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.155+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 shutdown_connections 2024-03-28T02:10:35.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.155+0000 7f62c5dc6640 1 -- 172.21.15.106:0/3805916451 wait complete. 2024-03-28T02:10:35.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:10:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.267+0000 7fa628cff640 1 -- 172.21.15.106:0/2878504223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa624102140 msgr2=0x7fa624104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.267+0000 7fa628cff640 1 --2- 172.21.15.106:0/2878504223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa624102140 0x7fa624104530 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009860 tx=0x7fa61c02bb70 comp rx=0 tx=0).stop 2024-03-28T02:10:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 -- 172.21.15.106:0/2878504223 shutdown_connections 2024-03-28T02:10:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 --2- 172.21.15.106:0/2878504223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6241073a0 0x7fa6241097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 --2- 172.21.15.106:0/2878504223 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa624104a70 0x7fa624106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 --2- 172.21.15.106:0/2878504223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa624102140 0x7fa624104530 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 -- 172.21.15.106:0/2878504223 >> 172.21.15.106:0/2878504223 conn(0x7fa6240fbb30 msgr2=0x7fa6240fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 -- 172.21.15.106:0/2878504223 shutdown_connections 2024-03-28T02:10:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 -- 172.21.15.106:0/2878504223 wait complete. 2024-03-28T02:10:35.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 Processor -- start 2024-03-28T02:10:35.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 -- start start 2024-03-28T02:10:35.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa624102140 0x7fa624197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa624104a70 0x7fa624197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6241073a0 0x7fa62419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62410d350 con 0x7fa6241073a0 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62410d1d0 con 0x7fa624104a70 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.268+0000 7fa628cff640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62410d4d0 con 0x7fa624102140 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa622d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6241073a0 0x7fa62419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa62259c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa624102140 0x7fa624197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa621d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa624104a70 0x7fa624197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa622d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6241073a0 0x7fa62419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56064/0 (socket says 172.21.15.106:56064) 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa62259c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa624102140 0x7fa624197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:40158/0 (socket says 172.21.15.106:40158) 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa622d9d640 1 -- 172.21.15.106:0/1176864489 learned_addr learned my addr 172.21.15.106:0/1176864489 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa622d9d640 1 -- 172.21.15.106:0/1176864489 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa624102140 msgr2=0x7fa624197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa622d9d640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa624102140 0x7fa624197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa622d9d640 1 -- 172.21.15.106:0/1176864489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa624104a70 msgr2=0x7fa624197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa622d9d640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa624104a70 0x7fa624197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa622d9d640 1 -- 172.21.15.106:0/1176864489 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62419f670 con 0x7fa6241073a0 2024-03-28T02:10:35.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa622d9d640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6241073a0 0x7fa62419f000 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa61400e940 tx=0x7fa61400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:35.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa6137fe640 1 -- 172.21.15.106:0/1176864489 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61400cd80 con 0x7fa6241073a0 2024-03-28T02:10:35.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62419f960 con 0x7fa6241073a0 2024-03-28T02:10:35.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa6137fe640 1 -- 172.21.15.106:0/1176864489 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6140035f0 con 0x7fa6241073a0 2024-03-28T02:10:35.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa6137fe640 1 -- 172.21.15.106:0/1176864489 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa614010690 con 0x7fa6241073a0 2024-03-28T02:10:35.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.269+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62419fe60 con 0x7fa6241073a0 2024-03-28T02:10:35.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.270+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6240fd620 con 0x7fa6241073a0 2024-03-28T02:10:35.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.271+0000 7fa6137fe640 1 -- 172.21.15.106:0/1176864489 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa6140037a0 con 0x7fa6241073a0 2024-03-28T02:10:35.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.272+0000 7fa6137fe640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa60c0a2800 0x7fa60c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.272+0000 7fa6137fe640 1 -- 172.21.15.106:0/1176864489 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fa6140cf170 con 0x7fa6241073a0 2024-03-28T02:10:35.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.272+0000 7fa62259c640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa60c0a2800 0x7fa60c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.272+0000 7fa62259c640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa60c0a2800 0x7fa60c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa61c02c180 tx=0x7fa61c034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:35.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.275+0000 7fa6137fe640 1 -- 172.21.15.106:0/1176864489 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa61409bf60 con 0x7fa6241073a0 2024-03-28T02:10:35.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.491+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa62402d070 con 0x7fa6241073a0 2024-03-28T02:10:35.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.491+0000 7fa6137fe640 1 -- 172.21.15.106:0/1176864489 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11474 (secure 0 0 0) 0x7fa61409b860 con 0x7fa6241073a0 2024-03-28T02:10:35.484 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:35.485 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:09:11.288485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24499,"mds_2":24460,"mds_3":24446,"mds_4":24415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14625":{"gid":14625,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14637":{"gid":14637,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24446":{"gid":24446,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24415,24499,24509,24446,24460]},"id":1}]} 2024-03-28T02:10:35.485 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 28 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa60c0a2800 msgr2=0x7fa60c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa60c0a2800 0x7fa60c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa61c02c180 tx=0x7fa61c034ad0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6241073a0 msgr2=0x7fa62419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6241073a0 0x7fa62419f000 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa61400e940 tx=0x7fa61400ee00 comp rx=0 tx=0).stop 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 shutdown_connections 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6241073a0 0x7fa62419f000 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa60c0a2800 0x7fa60c0a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa624104a70 0x7fa624197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 --2- 172.21.15.106:0/1176864489 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa624102140 0x7fa624197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 >> 172.21.15.106:0/1176864489 conn(0x7fa6240fbb30 msgr2=0x7fa624102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 shutdown_connections 2024-03-28T02:10:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.493+0000 7fa628cff640 1 -- 172.21.15.106:0/1176864489 wait complete. 2024-03-28T02:10:35.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:10:35.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.607+0000 7fcd954f8640 1 -- 172.21.15.106:0/2671077550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901017b0 msgr2=0x7fcd90101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.607+0000 7fcd954f8640 1 --2- 172.21.15.106:0/2671077550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901017b0 0x7fcd90101c30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcd78009900 tx=0x7fcd7802bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:35.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.607+0000 7fcd954f8640 1 -- 172.21.15.106:0/2671077550 shutdown_connections 2024-03-28T02:10:35.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.607+0000 7fcd954f8640 1 --2- 172.21.15.106:0/2671077550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd90102170 0x7fcd90108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.607+0000 7fcd954f8640 1 --2- 172.21.15.106:0/2671077550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901017b0 0x7fcd90101c30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.607+0000 7fcd954f8640 1 --2- 172.21.15.106:0/2671077550 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd901005c0 0x7fcd901009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.607+0000 7fcd954f8640 1 -- 172.21.15.106:0/2671077550 >> 172.21.15.106:0/2671077550 conn(0x7fcd900fbb50 msgr2=0x7fcd900fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:35.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.607+0000 7fcd954f8640 1 -- 172.21.15.106:0/2671077550 shutdown_connections 2024-03-28T02:10:35.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.607+0000 7fcd954f8640 1 -- 172.21.15.106:0/2671077550 wait complete. 2024-03-28T02:10:35.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd954f8640 1 Processor -- start 2024-03-28T02:10:35.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd954f8640 1 -- start start 2024-03-28T02:10:35.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd954f8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901005c0 0x7fcd9006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd954f8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd901017b0 0x7fcd9019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd954f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd90102170 0x7fcd9019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd954f8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9010c240 con 0x7fcd901005c0 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd954f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9010c0c0 con 0x7fcd90102170 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd954f8640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9010c3c0 con 0x7fcd901017b0 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd8effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901005c0 0x7fcd9006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd8effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901005c0 0x7fcd9006a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56086/0 (socket says 172.21.15.106:56086) 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd8effd640 1 -- 172.21.15.106:0/1069005852 learned_addr learned my addr 172.21.15.106:0/1069005852 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd867fc640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd901017b0 0x7fcd9019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd8f7fe640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd90102170 0x7fcd9019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd8effd640 1 -- 172.21.15.106:0/1069005852 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd901017b0 msgr2=0x7fcd9019c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd8effd640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd901017b0 0x7fcd9019c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd8effd640 1 -- 172.21.15.106:0/1069005852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd90102170 msgr2=0x7fcd9019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd8effd640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd90102170 0x7fcd9019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.608+0000 7fcd8effd640 1 -- 172.21.15.106:0/1069005852 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9019f610 con 0x7fcd901005c0 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.609+0000 7fcd8effd640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901005c0 0x7fcd9006a8c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcd8000e9c0 tx=0x7fcd8000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:35.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.609+0000 7fcd8cff9640 1 -- 172.21.15.106:0/1069005852 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8000cd80 con 0x7fcd901005c0 2024-03-28T02:10:35.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.609+0000 7fcd8cff9640 1 -- 172.21.15.106:0/1069005852 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd800035f0 con 0x7fcd901005c0 2024-03-28T02:10:35.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.609+0000 7fcd8cff9640 1 -- 172.21.15.106:0/1069005852 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd80010690 con 0x7fcd901005c0 2024-03-28T02:10:35.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.609+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9019f900 con 0x7fcd901005c0 2024-03-28T02:10:35.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.609+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9019fdd0 con 0x7fcd901005c0 2024-03-28T02:10:35.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.610+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd901078e0 con 0x7fcd901005c0 2024-03-28T02:10:35.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.611+0000 7fcd8cff9640 1 -- 172.21.15.106:0/1069005852 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcd800037a0 con 0x7fcd901005c0 2024-03-28T02:10:35.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.611+0000 7fcd8cff9640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd7c0a2800 0x7fcd7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.611+0000 7fcd867fc640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd7c0a2800 0x7fcd7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.612+0000 7fcd8cff9640 1 -- 172.21.15.106:0/1069005852 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fcd800cf100 con 0x7fcd901005c0 2024-03-28T02:10:35.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.612+0000 7fcd867fc640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd7c0a2800 0x7fcd7c0a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcd7802c7b0 tx=0x7fcd78005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:35.608 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.616+0000 7fcd8cff9640 1 -- 172.21.15.106:0/1069005852 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcd8009be60 con 0x7fcd901005c0 2024-03-28T02:10:35.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.793+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd900622a0 con 0x7fcd901005c0 2024-03-28T02:10:35.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.795+0000 7fcd8cff9640 1 -- 172.21.15.106:0/1069005852 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24705 (secure 0 0 0) 0x7fcd8009b760 con 0x7fcd901005c0 2024-03-28T02:10:35.788 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:35.789 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":91,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:09:43.454083+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:10:35.790 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd7c0a2800 msgr2=0x7fcd7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd7c0a2800 0x7fcd7c0a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcd7802c7b0 tx=0x7fcd78005fd0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901005c0 msgr2=0x7fcd9006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901005c0 0x7fcd9006a8c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcd8000e9c0 tx=0x7fcd8000ee80 comp rx=0 tx=0).stop 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 shutdown_connections 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcd7c0a2800 0x7fcd7c0a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd90102170 0x7fcd9019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd901017b0 0x7fcd9019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 --2- 172.21.15.106:0/1069005852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd901005c0 0x7fcd9006a8c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 >> 172.21.15.106:0/1069005852 conn(0x7fcd900fbb50 msgr2=0x7fcd90104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 shutdown_connections 2024-03-28T02:10:35.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.797+0000 7fcd954f8640 1 -- 172.21.15.106:0/1069005852 wait complete. 2024-03-28T02:10:35.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:35 smithi106 ceph-mon[31965]: pgmap v347: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:35.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:35 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3316455708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:10:35.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:35 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3805916451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:10:35.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:35 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1176864489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:10:35.802 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:10:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:35 smithi178 ceph-mon[36521]: pgmap v347: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:35 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3316455708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:10:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:35 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3805916451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:10:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:35 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1176864489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 -- 172.21.15.106:0/2490660920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 msgr2=0x7f66d8108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 --2- 172.21.15.106:0/2490660920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 0x7f66d8108a00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009900 tx=0x7f66c002bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 -- 172.21.15.106:0/2490660920 shutdown_connections 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 --2- 172.21.15.106:0/2490660920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 0x7f66d8108a00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 --2- 172.21.15.106:0/2490660920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d81017b0 0x7f66d8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 --2- 172.21.15.106:0/2490660920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d81005c0 0x7f66d81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 -- 172.21.15.106:0/2490660920 >> 172.21.15.106:0/2490660920 conn(0x7f66d80fbb30 msgr2=0x7f66d80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 -- 172.21.15.106:0/2490660920 shutdown_connections 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 -- 172.21.15.106:0/2490660920 wait complete. 2024-03-28T02:10:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 Processor -- start 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 -- start start 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d81005c0 0x7f66d8195730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d81017b0 0x7f66d8195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 0x7f66d819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d810b140 con 0x7f66d8102170 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d810afc0 con 0x7f66d81005c0 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.911+0000 7f66dee4b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d810b2c0 con 0x7f66d81017b0 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dd3e8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 0x7f66d819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dcbe7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d81005c0 0x7f66d8195730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dd3e8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 0x7f66d819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56102/0 (socket says 172.21.15.106:56102) 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dd3e8640 1 -- 172.21.15.106:0/4186478894 learned_addr learned my addr 172.21.15.106:0/4186478894 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66d7fff640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d81017b0 0x7f66d8195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dcbe7640 1 -- 172.21.15.106:0/4186478894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d81017b0 msgr2=0x7f66d8195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dcbe7640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d81017b0 0x7f66d8195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dcbe7640 1 -- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 msgr2=0x7f66d819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:35.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dcbe7640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 0x7f66d819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:35.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dcbe7640 1 -- 172.21.15.106:0/4186478894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d819d350 con 0x7f66d81005c0 2024-03-28T02:10:35.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dd3e8640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 0x7f66d819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:35.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dcbe7640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d81005c0 0x7f66d8195730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009aa0 tx=0x7f66d0009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:35.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66d5ffb640 1 -- 172.21.15.106:0/4186478894 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d001e070 con 0x7f66d81005c0 2024-03-28T02:10:35.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d819d640 con 0x7f66d81005c0 2024-03-28T02:10:35.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66d5ffb640 1 -- 172.21.15.106:0/4186478894 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66d0007d60 con 0x7f66d81005c0 2024-03-28T02:10:35.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.912+0000 7f66d5ffb640 1 -- 172.21.15.106:0/4186478894 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d0019650 con 0x7f66d81005c0 2024-03-28T02:10:35.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.913+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d819db40 con 0x7f66d81005c0 2024-03-28T02:10:35.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.913+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d81078e0 con 0x7f66d81005c0 2024-03-28T02:10:35.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.914+0000 7f66d5ffb640 1 -- 172.21.15.106:0/4186478894 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f66d0021070 con 0x7f66d81005c0 2024-03-28T02:10:35.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.915+0000 7f66d5ffb640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66c40a2800 0x7f66c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:35.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.915+0000 7f66d7fff640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66c40a2800 0x7f66c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:35.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.915+0000 7f66d5ffb640 1 -- 172.21.15.106:0/4186478894 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f66d00d2370 con 0x7f66d81005c0 2024-03-28T02:10:35.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.915+0000 7f66d7fff640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66c40a2800 0x7f66c40a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f66d8196c40 tx=0x7f66c800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:35.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:35.921+0000 7f66d5ffb640 1 -- 172.21.15.106:0/4186478894 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f66d0016070 con 0x7f66d81005c0 2024-03-28T02:10:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:35 smithi153 ceph-mon[36228]: pgmap v347: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:35 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3316455708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:10:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:35 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3805916451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:10:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:35 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1176864489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:10:36.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.132+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66d802d070 con 0x7f66d81005c0 2024-03-28T02:10:36.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.133+0000 7f66d5ffb640 1 -- 172.21.15.106:0/4186478894 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11474 (secure 0 0 0) 0x7f66d009eac0 con 0x7f66d81005c0 2024-03-28T02:10:36.126 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:36.126 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:09:11.288485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24499,"mds_2":24460,"mds_3":24446,"mds_4":24415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14625":{"gid":14625,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14637":{"gid":14637,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24446":{"gid":24446,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24415,24499,24509,24446,24460]},"id":1}]} 2024-03-28T02:10:36.126 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 28 2024-03-28T02:10:36.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66c40a2800 msgr2=0x7f66c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66c40a2800 0x7f66c40a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f66d8196c40 tx=0x7f66c800d040 comp rx=0 tx=0).stop 2024-03-28T02:10:36.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d81005c0 msgr2=0x7f66d8195730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d81005c0 0x7f66d8195730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009aa0 tx=0x7f66d0009f60 comp rx=0 tx=0).stop 2024-03-28T02:10:36.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 shutdown_connections 2024-03-28T02:10:36.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d8102170 0x7f66d819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d81017b0 0x7f66d8195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f66c40a2800 0x7f66c40a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 --2- 172.21.15.106:0/4186478894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d81005c0 0x7f66d8195730 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 >> 172.21.15.106:0/4186478894 conn(0x7f66d80fbb30 msgr2=0x7f66d8104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 shutdown_connections 2024-03-28T02:10:36.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.135+0000 7f66dee4b640 1 -- 172.21.15.106:0/4186478894 wait complete. 2024-03-28T02:10:36.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T02:08:48.782692+0000', 'modified': '2024-03-28T02:09:11.288485+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24509, 'mds_1': 24499, 'mds_2': 24460, 'mds_3': 24446, 'mds_4': 24415}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14625': {'gid': 14625, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6839/1515774363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6838', 'nonce': 1515774363}, {'type': 'v1', 'addr': '172.21.15.106:6839', 'nonce': 1515774363}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14637': {'gid': 14637, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6841/1766684450', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6840', 'nonce': 1766684450}, {'type': 'v1', 'addr': '172.21.15.106:6841', 'nonce': 1766684450}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24404': {'gid': 24404, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6833/564442660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6832', 'nonce': 564442660}, {'type': 'v1', 'addr': '172.21.15.153:6833', 'nonce': 564442660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24415': {'gid': 24415, 'name': 'l', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.178:6833/1496777187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6832', 'nonce': 1496777187}, {'type': 'v1', 'addr': '172.21.15.178:6833', 'nonce': 1496777187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24446': {'gid': 24446, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.153:6835/3469968496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 3469968496}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 3469968496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.178:6835/1402366472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6834', 'nonce': 1402366472}, {'type': 'v1', 'addr': '172.21.15.178:6835', 'nonce': 1402366472}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6837/3554528514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6836', 'nonce': 3554528514}, {'type': 'v1', 'addr': '172.21.15.178:6837', 'nonce': 3554528514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6837/1253460068', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 1253460068}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 1253460068}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.178:6839/937216083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6838', 'nonce': 937216083}, {'type': 'v1', 'addr': '172.21.15.178:6839', 'nonce': 937216083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6839/505906484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 505906484}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 505906484}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24509, 'qdb_cluster': [24415, 24499, 24509, 24446, 24460]} 2024-03-28T02:10:36.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T02:10:36.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-03-28T02:10:36.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.249+0000 7f88e6419640 1 -- 172.21.15.106:0/3755550053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e0102170 msgr2=0x7f88e0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.249+0000 7f88e6419640 1 --2- 172.21.15.106:0/3755550053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e0102170 0x7f88e0108a00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009ca0 tx=0x7f88d802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.249+0000 7f88e6419640 1 -- 172.21.15.106:0/3755550053 shutdown_connections 2024-03-28T02:10:36.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.249+0000 7f88e6419640 1 --2- 172.21.15.106:0/3755550053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e0102170 0x7f88e0108a00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.249+0000 7f88e6419640 1 --2- 172.21.15.106:0/3755550053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88e01017b0 0x7f88e0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.249+0000 7f88e6419640 1 --2- 172.21.15.106:0/3755550053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88e01005c0 0x7f88e01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.249+0000 7f88e6419640 1 -- 172.21.15.106:0/3755550053 >> 172.21.15.106:0/3755550053 conn(0x7f88e00fbb50 msgr2=0x7f88e00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.249+0000 7f88e6419640 1 -- 172.21.15.106:0/3755550053 shutdown_connections 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.249+0000 7f88e6419640 1 -- 172.21.15.106:0/3755550053 wait complete. 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88e6419640 1 Processor -- start 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88e6419640 1 -- start start 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88e6419640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88e01005c0 0x7f88e006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88e6419640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e01017b0 0x7f88e019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88e6419640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88e0102170 0x7f88e019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88e6419640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e010c240 con 0x7f88e01017b0 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88e6419640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e010c0c0 con 0x7f88e0102170 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88e6419640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e010c3c0 con 0x7f88e01005c0 2024-03-28T02:10:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88df7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e01017b0 0x7f88e019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88df7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e01017b0 0x7f88e019c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56128/0 (socket says 172.21.15.106:56128) 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88df7fe640 1 -- 172.21.15.106:0/2549757853 learned_addr learned my addr 172.21.15.106:0/2549757853 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88dffff640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88e01005c0 0x7f88e006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.250+0000 7f88e49b6640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88e0102170 0x7f88e019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88df7fe640 1 -- 172.21.15.106:0/2549757853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88e01005c0 msgr2=0x7f88e006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88e01005c0 0x7f88e006a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88df7fe640 1 -- 172.21.15.106:0/2549757853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88e0102170 msgr2=0x7f88e019ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88e0102170 0x7f88e019ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88df7fe640 1 -- 172.21.15.106:0/2549757853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e019f610 con 0x7f88e01017b0 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e01017b0 0x7f88e019c6e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f88d400e9f0 tx=0x7f88d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88dd7fa640 1 -- 172.21.15.106:0/2549757853 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d400cd80 con 0x7f88e01017b0 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88dd7fa640 1 -- 172.21.15.106:0/2549757853 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88d4005500 con 0x7f88e01017b0 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88e6419640 1 -- 172.21.15.106:0/2549757853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e019f8a0 con 0x7f88e01017b0 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88dd7fa640 1 -- 172.21.15.106:0/2549757853 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d4010690 con 0x7f88e01017b0 2024-03-28T02:10:36.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.251+0000 7f88e6419640 1 -- 172.21.15.106:0/2549757853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e019fda0 con 0x7f88e01017b0 2024-03-28T02:10:36.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.253+0000 7f88dd7fa640 1 -- 172.21.15.106:0/2549757853 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f88d40046c0 con 0x7f88e01017b0 2024-03-28T02:10:36.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e9008740 1 Processor -- start 2024-03-28T02:10:36.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e9008740 1 -- start start 2024-03-28T02:10:36.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e9008740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 0x5613a5ae8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e9008740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5aedc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e9008740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5aee2a0 0x5613a5af0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e9008740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5a4dc20 con 0x5613a5ae9130 2024-03-28T02:10:36.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e9008740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a59f6a40 con 0x5613a5ae8740 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e9008740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5a49cc0 con 0x5613a5aee2a0 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dd7fa640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88c80a2800 0x7f88c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5aedc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e49b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 0x5613a5ae8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5aedc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56144/0 (socket says 172.21.15.106:56144) 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dffff640 1 -- 172.21.15.106:0/2147170692 learned_addr learned my addr 172.21.15.106:0/2147170692 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2147170692 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5aee2a0 0x5613a5af0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88e49b6640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88c80a2800 0x7f88c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dd7fa640 1 -- 172.21.15.106:0/2549757853 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f88d40cf190 con 0x7f88e01017b0 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dffff640 1 -- 172.21.15.106:0/2147170692 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5aee2a0 msgr2=0x5613a5af0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dffff640 1 --2- 172.21.15.106:0/2147170692 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5aee2a0 0x5613a5af0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dffff640 1 -- 172.21.15.106:0/2147170692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 msgr2=0x5613a5ae8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dffff640 1 --2- 172.21.15.106:0/2147170692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 0x5613a5ae8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.254+0000 7f88dffff640 1 -- 172.21.15.106:0/2147170692 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5613a5a18f90 con 0x5613a5ae9130 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88e49b6640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88c80a2800 0x7f88c80a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f88d8005e30 tx=0x7f88d8032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88dffff640 1 --2- 172.21.15.106:0/2147170692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5aedc30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f88d0009ca0 tx=0x7f88d0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f4c984e9e30b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88c67fc640 1 -- 172.21.15.106:0/2147170692 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d0004220 con 0x5613a5ae9130 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88c67fc640 1 -- 172.21.15.106:0/2147170692 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88d00043c0 con 0x5613a5ae9130 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88c67fc640 1 -- 172.21.15.106:0/2147170692 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d00198d0 con 0x5613a5ae9130 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88e9008740 1 -- 172.21.15.106:0/2147170692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 msgr2=0x5613a5aedc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88e9008740 1 --2- 172.21.15.106:0/2147170692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5aedc30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f88d0009ca0 tx=0x7f88d0011810 comp rx=0 tx=0).stop 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88e9008740 1 -- 172.21.15.106:0/2147170692 shutdown_connections 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88e9008740 1 --2- 172.21.15.106:0/2147170692 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5aee2a0 0x5613a5af0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88e9008740 1 --2- 172.21.15.106:0/2147170692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5aedc30 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88e9008740 1 --2- 172.21.15.106:0/2147170692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 0x5613a5ae8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88e9008740 1 -- 172.21.15.106:0/2147170692 >> 172.21.15.106:0/2147170692 conn(0x5613a5ab0a60 msgr2=0x5613a5ae40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.255+0000 7f88e9008740 1 -- 172.21.15.106:0/2147170692 shutdown_connections 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 1 -- 172.21.15.106:0/2147170692 wait complete. 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 1 -- start start 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 1 Processor -- start 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae8740 0x5613a5af2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5af3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5aee2a0 0x5613a5afa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5af1250 con 0x5613a5ae9130 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5a09e90 con 0x5613a5aee2a0 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5af13d0 con 0x5613a5ae8740 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e9008740 10 client.0 _ll_register_callbacks cb 0x5613a5761ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88df7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5af3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88df7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5af3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56146/0 (socket says 172.21.15.106:56146) 2024-03-28T02:10:36.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88df7fe640 1 -- 172.21.15.106:0/2643367607 learned_addr learned my addr 172.21.15.106:0/2643367607 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88dffff640 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae8740 0x5613a5af2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.256+0000 7f88e49b6640 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5aee2a0 0x5613a5afa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88df7fe640 1 -- 172.21.15.106:0/2643367607 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae8740 msgr2=0x5613a5af2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae8740 0x5613a5af2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88df7fe640 1 -- 172.21.15.106:0/2643367607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5aee2a0 msgr2=0x5613a5afa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5aee2a0 0x5613a5afa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88df7fe640 1 -- 172.21.15.106:0/2643367607 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5613a5a169d0 con 0x5613a5ae9130 2024-03-28T02:10:36.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5af3130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f88d400c6a0 tx=0x7f88d400aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7677e0259aad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:10:36.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2643367607 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d409b150 con 0x5613a5ae9130 2024-03-28T02:10:36.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2643367607 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f88d40a3b00 con 0x5613a5ae9130 2024-03-28T02:10:36.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88e9008740 1 -- 172.21.15.106:0/2643367607 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88d00043c0 con 0x5613a5ae9130 2024-03-28T02:10:36.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2643367607 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88d409bef0 con 0x5613a5ae9130 2024-03-28T02:10:36.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88e9008740 10 client.15228 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88e9008740 1 -- 172.21.15.106:0/2643367607 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5613a57622d0 con 0x5613a5ae9130 2024-03-28T02:10:36.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2643367607 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88d00043c0 con 0x5613a5ae9130 2024-03-28T02:10:36.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2643367607 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f88d40bb050 con 0x5613a5ae9130 2024-03-28T02:10:36.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88e9008740 10 client.15228 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88e9008740 10 client.15228 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-28T02:10:36.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88e9008740 20 client.15228 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88e9008740 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x5613a5aff440 0x5613a5afe510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88e9008740 4 client.15228 mds_command: new command op to 24415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:10:36.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.257+0000 7f88e9008740 1 -- 172.21.15.106:0/2643367607 --> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5613a5a5b160 con 0x5613a5aff440 2024-03-28T02:10:36.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.258+0000 7f88dffff640 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x5613a5aff440 0x5613a5afe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.258+0000 7f88bf7fe640 10 client.15228 ms_handle_connect on v2:172.21.15.178:6832/1496777187 2024-03-28T02:10:36.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.258+0000 7f88dffff640 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x5613a5aff440 0x5613a5afe510 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=96c36d78158472e0 server_cookie=bb5419ca2f84e161 in_seq=0 out_seq=0 2024-03-28T02:10:36.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2643367607 <== mds.4 v2:172.21.15.178:6832/1496777187 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5613a5a5b160 con 0x5613a5aff440 2024-03-28T02:10:36.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88bf7fe640 10 client.15228 handle_command_reply: tid=0 2024-03-28T02:10:36.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 client.15228 shutdown 2024-03-28T02:10:36.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 -- 172.21.15.106:0/2643367607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 msgr2=0x5613a5af3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5af3130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f88d400c6a0 tx=0x7f88d400aea0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 -- 172.21.15.106:0/2643367607 shutdown_connections 2024-03-28T02:10:36.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x5613a5aff440 0x5613a5afe510 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5aee2a0 0x5613a5afa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88bf7fe640 0 client.15228 ms_handle_reset on v2:172.21.15.178:6832/1496777187 2024-03-28T02:10:36.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae9130 0x5613a5af3130 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 --2- 172.21.15.106:0/2643367607 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae8740 0x5613a5af2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 -- 172.21.15.106:0/2643367607 >> 172.21.15.106:0/2643367607 conn(0x5613a5ab0a60 msgr2=0x5613a5ae32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 -- 172.21.15.106:0/2643367607 shutdown_connections 2024-03-28T02:10:36.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 1 -- 172.21.15.106:0/2643367607 wait complete. 2024-03-28T02:10:36.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.259+0000 7f88e9008740 20 client.15228 trim_cache size 0 max 16384 2024-03-28T02:10:36.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e9008740 1 Processor -- start 2024-03-28T02:10:36.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e9008740 1 -- start start 2024-03-28T02:10:36.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e9008740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ab0a60 0x5613a5af17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e9008740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 0x5613a5afa130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e9008740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae9130 0x5613a5aea590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e9008740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5afec60 con 0x5613a5ab0a60 2024-03-28T02:10:36.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e9008740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5aff2c0 con 0x5613a5ae8740 2024-03-28T02:10:36.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e9008740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5aff920 con 0x5613a5ae9130 2024-03-28T02:10:36.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88dffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ab0a60 0x5613a5af17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88dffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ab0a60 0x5613a5af17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56156/0 (socket says 172.21.15.106:56156) 2024-03-28T02:10:36.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88dffff640 1 -- 172.21.15.106:0/1838113565 learned_addr learned my addr 172.21.15.106:0/1838113565 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e49b6640 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae9130 0x5613a5aea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88df7fe640 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 0x5613a5afa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88dffff640 1 -- 172.21.15.106:0/1838113565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae9130 msgr2=0x5613a5aea590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88dffff640 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae9130 0x5613a5aea590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88dffff640 1 -- 172.21.15.106:0/1838113565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 msgr2=0x5613a5afa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88dffff640 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 0x5613a5afa130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88dffff640 1 -- 172.21.15.106:0/1838113565 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5613a582c6b0 con 0x5613a5ab0a60 2024-03-28T02:10:36.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88e49b6640 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae9130 0x5613a5aea590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:10:36.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88dffff640 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ab0a60 0x5613a5af17d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f88d00040c0 tx=0x7f88d0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d08645b6fe1352 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88c5ffb640 1 -- 172.21.15.106:0/1838113565 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d002f020 con 0x5613a5ab0a60 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88c5ffb640 1 -- 172.21.15.106:0/1838113565 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88d0005520 con 0x5613a5ab0a60 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.276+0000 7f88c5ffb640 1 -- 172.21.15.106:0/1838113565 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d0003ae0 con 0x5613a5ab0a60 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 -- 172.21.15.106:0/1838113565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ab0a60 msgr2=0x5613a5af17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ab0a60 0x5613a5af17d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f88d00040c0 tx=0x7f88d0011810 comp rx=0 tx=0).stop 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 -- 172.21.15.106:0/1838113565 shutdown_connections 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ae9130 0x5613a5aea590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae8740 0x5613a5afa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 --2- 172.21.15.106:0/1838113565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ab0a60 0x5613a5af17d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 -- 172.21.15.106:0/1838113565 >> 172.21.15.106:0/1838113565 conn(0x5613a5afe880 msgr2=0x5613a5af2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 -- 172.21.15.106:0/1838113565 shutdown_connections 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 -- 172.21.15.106:0/1838113565 wait complete. 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 -- start start 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.277+0000 7f88e9008740 1 Processor -- start 2024-03-28T02:10:36.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e9008740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ab0a60 0x5613a5af99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e9008740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae8740 0x5613a5af9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e9008740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae9130 0x5613a5af6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e9008740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5b005e0 con 0x5613a5ae8740 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e9008740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5afff80 con 0x5613a5ae9130 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e9008740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613a5b00c40 con 0x5613a5ab0a60 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e9008740 10 client.0 _ll_register_callbacks cb 0x7f88d400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88df7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae8740 0x5613a5af9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88df7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae8740 0x5613a5af9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56160/0 (socket says 172.21.15.106:56160) 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88df7fe640 1 -- 172.21.15.106:0/2054783082 learned_addr learned my addr 172.21.15.106:0/2054783082 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88dffff640 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ab0a60 0x5613a5af99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e49b6640 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae9130 0x5613a5af6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88df7fe640 1 -- 172.21.15.106:0/2054783082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ab0a60 msgr2=0x5613a5af99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ab0a60 0x5613a5af99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88df7fe640 1 -- 172.21.15.106:0/2054783082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae9130 msgr2=0x5613a5af6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae9130 0x5613a5af6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88df7fe640 1 -- 172.21.15.106:0/2054783082 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5613a5af6b20 con 0x5613a5ae8740 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e49b6640 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae9130 0x5613a5af6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88df7fe640 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae8740 0x5613a5af9f00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f88d40097d0 tx=0x7f88d4005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede71c29083ae504 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2054783082 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d40b8070 con 0x5613a5ae8740 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2054783082 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f88d409ac40 con 0x5613a5ae8740 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.278+0000 7f88e9008740 1 -- 172.21.15.106:0/2054783082 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88d0005520 con 0x5613a5ae8740 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2054783082 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88d409b150 con 0x5613a5ae8740 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88e9008740 10 client.15234 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88e9008740 1 -- 172.21.15.106:0/2054783082 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5613a5b5b110 con 0x5613a5ae8740 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2054783082 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88d0005520 con 0x5613a5ae8740 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2054783082 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f88d40bb040 con 0x5613a5ae8740 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88e9008740 10 client.15234 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88e9008740 10 client.15234 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-28T02:10:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88e9008740 20 client.15234 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88e9008740 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x5613a5aee2a0 0x5613a5b5b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88e9008740 4 client.15234 mds_command: new command op to 24415 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88e9008740 1 -- 172.21.15.106:0/2054783082 --> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5613a5a554b0 con 0x5613a5aee2a0 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88dffff640 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x5613a5aee2a0 0x5613a5b5b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88dffff640 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x5613a5aee2a0 0x5613a5b5b3a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=49ad64d506b976be server_cookie=2d7ba8d46d243ce1 in_seq=0 out_seq=0 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.279+0000 7f88bf7fe640 10 client.15234 ms_handle_connect on v2:172.21.15.178:6832/1496777187 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88bf7fe640 1 -- 172.21.15.106:0/2054783082 <== mds.4 v2:172.21.15.178:6832/1496777187 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5613a5a554b0 con 0x5613a5aee2a0 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88bf7fe640 10 client.15234 handle_command_reply: tid=0 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 client.15234 shutdown 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 -- 172.21.15.106:0/2054783082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae8740 msgr2=0x5613a5af9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae8740 0x5613a5af9f00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f88d40097d0 tx=0x7f88d4005730 comp rx=0 tx=0).stop 2024-03-28T02:10:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 -- 172.21.15.106:0/2054783082 shutdown_connections 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x5613a5aee2a0 0x5613a5b5b3a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613a5ae9130 0x5613a5af6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5613a5ae8740 0x5613a5af9f00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88bf7fe640 0 client.15234 ms_handle_reset on v2:172.21.15.178:6832/1496777187 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 --2- 172.21.15.106:0/2054783082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5613a5ab0a60 0x5613a5af99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 -- 172.21.15.106:0/2054783082 >> 172.21.15.106:0/2054783082 conn(0x5613a5afe880 msgr2=0x5613a5af17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 -- 172.21.15.106:0/2054783082 shutdown_connections 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 1 -- 172.21.15.106:0/2054783082 wait complete. 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e9008740 20 client.15234 trim_cache size 0 max 16384 2024-03-28T02:10:36.287 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 4, 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24415, 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 86.011564113000006, 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 91, 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 145.23383551399999 2024-03-28T02:10:36.288 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:10:36.289 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e6419640 1 -- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88c80a2800 msgr2=0x7f88c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e6419640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88c80a2800 0x7f88c80a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f88d8005e30 tx=0x7f88d8032590 comp rx=0 tx=0).stop 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e6419640 1 -- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e01017b0 msgr2=0x7f88e019c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.280+0000 7f88e6419640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e01017b0 0x7f88e019c6e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f88d400e9f0 tx=0x7f88d400edf0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.281+0000 7f88e6419640 1 -- 172.21.15.106:0/2549757853 shutdown_connections 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.281+0000 7f88e6419640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f88c80a2800 0x7f88c80a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.281+0000 7f88e6419640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88e0102170 0x7f88e019ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.281+0000 7f88e6419640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88e01017b0 0x7f88e019c6e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.281+0000 7f88e6419640 1 --2- 172.21.15.106:0/2549757853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88e01005c0 0x7f88e006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.281+0000 7f88e6419640 1 -- 172.21.15.106:0/2549757853 >> 172.21.15.106:0/2549757853 conn(0x7f88e00fbb50 msgr2=0x7f88e0104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.281+0000 7f88e6419640 1 -- 172.21.15.106:0/2549757853 shutdown_connections 2024-03-28T02:10:36.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.281+0000 7f88e6419640 1 -- 172.21.15.106:0/2549757853 wait complete. 2024-03-28T02:10:36.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-28T02:10:36.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 -- 172.21.15.106:0/3376170495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681028a0 msgr2=0x7f4a68102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/3376170495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681028a0 0x7f4a68102d20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009900 tx=0x7f4a5c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:36.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 -- 172.21.15.106:0/3376170495 shutdown_connections 2024-03-28T02:10:36.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/3376170495 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a68103260 0x7f4a68109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/3376170495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681028a0 0x7f4a68102d20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/3376170495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a681016b0 0x7f4a68101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 -- 172.21.15.106:0/3376170495 >> 172.21.15.106:0/3376170495 conn(0x7f4a680fcc40 msgr2=0x7f4a680ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 -- 172.21.15.106:0/3376170495 shutdown_connections 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 -- 172.21.15.106:0/3376170495 wait complete. 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 Processor -- start 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 -- start start 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681016b0 0x7f4a68197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a681028a0 0x7f4a68197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.401+0000 7f4a6d99e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a68103260 0x7f4a6819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a6d99e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6810d310 con 0x7f4a68103260 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a6d99e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6810d190 con 0x7f4a681016b0 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a6d99e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6810d490 con 0x7f4a681028a0 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a677fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a68103260 0x7f4a6819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a66ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681016b0 0x7f4a68197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a677fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a68103260 0x7f4a6819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56180/0 (socket says 172.21.15.106:56180) 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a677fe640 1 -- 172.21.15.106:0/1272547175 learned_addr learned my addr 172.21.15.106:0/1272547175 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a66ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681016b0 0x7f4a68197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:38024/0 (socket says 172.21.15.106:38024) 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a667fc640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a681028a0 0x7f4a68197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a677fe640 1 -- 172.21.15.106:0/1272547175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a681028a0 msgr2=0x7f4a68197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a677fe640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a681028a0 0x7f4a68197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a677fe640 1 -- 172.21.15.106:0/1272547175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681016b0 msgr2=0x7f4a68197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a677fe640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681016b0 0x7f4a68197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a677fe640 1 -- 172.21.15.106:0/1272547175 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6819f630 con 0x7f4a68103260 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a66ffd640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681016b0 0x7f4a68197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a667fc640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a681028a0 0x7f4a68197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a677fe640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a68103260 0x7f4a6819efc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000efd0 tx=0x7f4a6000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a6c99c640 1 -- 172.21.15.106:0/1272547175 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60003b70 con 0x7f4a68103260 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a6d99e640 1 -- 172.21.15.106:0/1272547175 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6819f920 con 0x7f4a68103260 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a6c99c640 1 -- 172.21.15.106:0/1272547175 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a60003d10 con 0x7f4a68103260 2024-03-28T02:10:36.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a6c99c640 1 -- 172.21.15.106:0/1272547175 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600109f0 con 0x7f4a68103260 2024-03-28T02:10:36.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.402+0000 7f4a6d99e640 1 -- 172.21.15.106:0/1272547175 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6819fdf0 con 0x7f4a68103260 2024-03-28T02:10:36.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.404+0000 7f4a6c99c640 1 -- 172.21.15.106:0/1272547175 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4a60004020 con 0x7f4a68103260 2024-03-28T02:10:36.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a70588740 1 Processor -- start 2024-03-28T02:10:36.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a70588740 1 -- start start 2024-03-28T02:10:36.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a70588740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc045740 0x5602bc045b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a70588740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc04ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a70588740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc04b2a0 0x5602bc04d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a70588740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bbfaac20 con 0x5602bc045740 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a70588740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bbf53a40 con 0x5602bc046130 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a70588740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bbfa6cc0 con 0x5602bc04b2a0 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc045740 0x5602bc045b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc045740 0x5602bc045b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56192/0 (socket says 172.21.15.106:56192) 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 -- 172.21.15.106:0/2403432307 learned_addr learned my addr 172.21.15.106:0/2403432307 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a66ffd640 1 --2- 172.21.15.106:0/2403432307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc04ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a6c99c640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4a500a2800 0x7f4a500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 --2- 172.21.15.106:0/2403432307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc04b2a0 0x5602bc04d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 -- 172.21.15.106:0/2403432307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc04b2a0 msgr2=0x5602bc04d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a677fe640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4a500a2800 0x7f4a500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 --2- 172.21.15.106:0/2403432307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc04b2a0 0x5602bc04d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 -- 172.21.15.106:0/2403432307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 msgr2=0x5602bc04ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 --2- 172.21.15.106:0/2403432307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc04ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 -- 172.21.15.106:0/2403432307 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602bbf75f90 con 0x5602bc045740 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a6c99c640 1 -- 172.21.15.106:0/1272547175 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f4a600cf210 con 0x7f4a68103260 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.405+0000 7f4a667fc640 1 --2- 172.21.15.106:0/2403432307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc045740 0x5602bc045b60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c006010 tx=0x7f4a5c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1106c9a7655eb82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a4dffb640 1 -- 172.21.15.106:0/2403432307 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0042c0 con 0x5602bc045740 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a4dffb640 1 -- 172.21.15.106:0/2403432307 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c048070 con 0x5602bc045740 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a4dffb640 1 -- 172.21.15.106:0/2403432307 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c004c00 con 0x5602bc045740 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a70588740 1 -- 172.21.15.106:0/2403432307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc045740 msgr2=0x5602bc045b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a677fe640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4a500a2800 0x7f4a500a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4a6009b800 tx=0x7f4a60086040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a70588740 1 --2- 172.21.15.106:0/2403432307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc045740 0x5602bc045b60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c006010 tx=0x7f4a5c005b50 comp rx=0 tx=0).stop 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a70588740 1 -- 172.21.15.106:0/2403432307 shutdown_connections 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a70588740 1 --2- 172.21.15.106:0/2403432307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc04b2a0 0x5602bc04d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a70588740 1 --2- 172.21.15.106:0/2403432307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc04ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a70588740 1 --2- 172.21.15.106:0/2403432307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc045740 0x5602bc045b60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a70588740 1 -- 172.21.15.106:0/2403432307 >> 172.21.15.106:0/2403432307 conn(0x5602bc00da60 msgr2=0x5602bc0410b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a70588740 1 -- 172.21.15.106:0/2403432307 shutdown_connections 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.406+0000 7f4a70588740 1 -- 172.21.15.106:0/2403432307 wait complete. 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a70588740 1 -- start start 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a70588740 1 Processor -- start 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a70588740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc04fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a70588740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc050200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a70588740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0571d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a70588740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bc04e1f0 con 0x5602bc04b2a0 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a70588740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bbf66e90 con 0x5602bc046130 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a70588740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bc04e370 con 0x5602bc045740 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a70588740 10 client.0 _ll_register_callbacks cb 0x5602bbcbeac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a667fc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc04fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a667fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a667fc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc04fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:40302/0 (socket says 172.21.15.106:40302) 2024-03-28T02:10:36.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a667fc640 1 -- 172.21.15.106:0/2466672286 learned_addr learned my addr 172.21.15.106:0/2466672286 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.407+0000 7f4a66ffd640 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc050200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a667fc640 1 -- 172.21.15.106:0/2466672286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 msgr2=0x5602bc050200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a667fc640 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc050200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a667fc640 1 -- 172.21.15.106:0/2466672286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 msgr2=0x5602bc0571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a667fc640 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a667fc640 1 -- 172.21.15.106:0/2466672286 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602bbf739d0 con 0x5602bc045740 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a667fc640 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc04fcc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c045040 tx=0x7f4a5c005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15c4376a3313dc44 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.178:3300/0 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a46ffd640 1 -- 172.21.15.106:0/2466672286 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c057070 con 0x5602bc045740 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a70588740 1 -- 172.21.15.106:0/2466672286 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a5c048070 con 0x5602bc045740 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a46ffd640 1 -- 172.21.15.106:0/2466672286 <== mon.1 v2:172.21.15.178:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f4a5c005220 con 0x5602bc045740 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a46ffd640 1 -- 172.21.15.106:0/2466672286 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a5c060050 con 0x5602bc045740 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a70588740 10 client.25018 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a70588740 1 -- 172.21.15.106:0/2466672286 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5602bbcbf2d0 con 0x5602bc045740 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a46ffd640 1 -- 172.21.15.106:0/2466672286 <== mon.1 v2:172.21.15.178:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c048070 con 0x5602bc045740 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a46ffd640 1 -- 172.21.15.106:0/2466672286 <== mon.1 v2:172.21.15.178:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f4a5c05a020 con 0x5602bc045740 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.408+0000 7f4a70588740 10 client.25018 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.409+0000 7f4a70588740 10 client.25018 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.409+0000 7f4a70588740 20 client.25018 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.409+0000 7f4a70588740 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5602bc05b330 0x5602bc0603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.409+0000 7f4a70588740 4 client.25018 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.409+0000 7f4a70588740 1 -- 172.21.15.106:0/2466672286 --> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602bbfb8160 con 0x5602bc05b330 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.409+0000 7f4a66ffd640 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5602bc05b330 0x5602bc0603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.409+0000 7f4a66ffd640 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5602bc05b330 0x5602bc0603d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2b5ef4d1a0baa8d8 server_cookie=1a59dd2a1a270a78 in_seq=0 out_seq=0 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.409+0000 7f4a46ffd640 10 client.25018 ms_handle_connect on v2:172.21.15.153:6834/3469968496 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a46ffd640 1 -- 172.21.15.106:0/2466672286 <== mds.3 v2:172.21.15.153:6834/3469968496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5602bbfb8160 con 0x5602bc05b330 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a46ffd640 10 client.25018 handle_command_reply: tid=0 2024-03-28T02:10:36.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 client.25018 shutdown 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 -- 172.21.15.106:0/2466672286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 msgr2=0x5602bc04fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc04fcc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c045040 tx=0x7f4a5c005c90 comp rx=0 tx=0).stop 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 -- 172.21.15.106:0/2466672286 shutdown_connections 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5602bc05b330 0x5602bc0603d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a46ffd640 0 client.25018 ms_handle_reset on v2:172.21.15.153:6834/3469968496 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc050200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 --2- 172.21.15.106:0/2466672286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc04fcc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 -- 172.21.15.106:0/2466672286 >> 172.21.15.106:0/2466672286 conn(0x5602bc00da60 msgr2=0x5602bc0402f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 -- 172.21.15.106:0/2466672286 shutdown_connections 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 1 -- 172.21.15.106:0/2466672286 wait complete. 2024-03-28T02:10:36.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.410+0000 7f4a70588740 20 client.25018 trim_cache size 0 max 16384 2024-03-28T02:10:36.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.426+0000 7f4a70588740 1 Processor -- start 2024-03-28T02:10:36.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.426+0000 7f4a70588740 1 -- start start 2024-03-28T02:10:36.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a70588740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc065400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a70588740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc057190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a70588740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0476d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a70588740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bc05b770 con 0x5602bc04b2a0 2024-03-28T02:10:36.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a70588740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bc05bdd0 con 0x5602bc046130 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a70588740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bc05c430 con 0x5602bc045740 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0476d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56218/0 (socket says 172.21.15.106:56218) 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 -- 172.21.15.106:0/1750367673 learned_addr learned my addr 172.21.15.106:0/1750367673 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a66ffd640 1 --2- 172.21.15.106:0/1750367673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc057190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 --2- 172.21.15.106:0/1750367673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc065400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 -- 172.21.15.106:0/1750367673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 msgr2=0x5602bc065400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 --2- 172.21.15.106:0/1750367673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc065400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 -- 172.21.15.106:0/1750367673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 msgr2=0x5602bc057190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 --2- 172.21.15.106:0/1750367673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc057190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 -- 172.21.15.106:0/1750367673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602bbd896b0 con 0x5602bc04b2a0 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a667fc640 1 --2- 172.21.15.106:0/1750367673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0476d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c02c650 tx=0x7f4a5c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5292618ec679bd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a4d7fa640 1 -- 172.21.15.106:0/1750367673 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c057040 con 0x5602bc04b2a0 2024-03-28T02:10:36.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a4d7fa640 1 -- 172.21.15.106:0/1750367673 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c048040 con 0x5602bc04b2a0 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.427+0000 7f4a4d7fa640 1 -- 172.21.15.106:0/1750367673 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0053a0 con 0x5602bc04b2a0 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 -- 172.21.15.106:0/1750367673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 msgr2=0x5602bc0476d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 --2- 172.21.15.106:0/1750367673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0476d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c02c650 tx=0x7f4a5c005b50 comp rx=0 tx=0).stop 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 -- 172.21.15.106:0/1750367673 shutdown_connections 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 --2- 172.21.15.106:0/1750367673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc0476d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 --2- 172.21.15.106:0/1750367673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc046130 0x5602bc057190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 --2- 172.21.15.106:0/1750367673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc045740 0x5602bc065400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 -- 172.21.15.106:0/1750367673 >> 172.21.15.106:0/1750367673 conn(0x5602bc00da60 msgr2=0x5602bc04e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 -- 172.21.15.106:0/1750367673 shutdown_connections 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 -- 172.21.15.106:0/1750367673 wait complete. 2024-03-28T02:10:36.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 -- start start 2024-03-28T02:10:36.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.428+0000 7f4a70588740 1 Processor -- start 2024-03-28T02:10:36.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc045740 0x5602bc052a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc046130 0x5602bc052fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc055f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bc05d0f0 con 0x5602bc04b2a0 2024-03-28T02:10:36.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bc05ca90 con 0x5602bc045740 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bc05d750 con 0x5602bc046130 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 10 client.0 _ll_register_callbacks cb 0x7f4a58009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc055f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc055f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56222/0 (socket says 172.21.15.106:56222) 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 -- 172.21.15.106:0/12294168 learned_addr learned my addr 172.21.15.106:0/12294168 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc045740 0x5602bc052a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a66ffd640 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc046130 0x5602bc052fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 -- 172.21.15.106:0/12294168 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc046130 msgr2=0x5602bc052fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc046130 0x5602bc052fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 -- 172.21.15.106:0/12294168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc045740 msgr2=0x5602bc052a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc045740 0x5602bc052a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 -- 172.21.15.106:0/12294168 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602bc056560 con 0x5602bc04b2a0 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a667fc640 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc055f70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c000c00 tx=0x7f4a5c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc417fb68f4cb042 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a46ffd640 1 -- 172.21.15.106:0/12294168 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c057040 con 0x5602bc04b2a0 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 1 -- 172.21.15.106:0/12294168 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a5c048040 con 0x5602bc04b2a0 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a46ffd640 1 -- 172.21.15.106:0/12294168 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f4a5c034b10 con 0x5602bc04b2a0 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a46ffd640 1 -- 172.21.15.106:0/12294168 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a5c034d10 con 0x5602bc04b2a0 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 10 client.15252 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.429+0000 7f4a70588740 1 -- 172.21.15.106:0/12294168 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5602bc0b7690 con 0x5602bc04b2a0 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a46ffd640 1 -- 172.21.15.106:0/12294168 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c048040 con 0x5602bc04b2a0 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a46ffd640 1 -- 172.21.15.106:0/12294168 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f4a5c05a040 con 0x5602bc04b2a0 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a70588740 10 client.15252 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a70588740 10 client.15252 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a70588740 20 client.15252 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a70588740 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5602bc0bb920 0x5602bc0b7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a70588740 4 client.15252 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a70588740 1 -- 172.21.15.106:0/12294168 --> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5602bbfb24b0 con 0x5602bc0bb920 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a66ffd640 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5602bc0bb920 0x5602bc0b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a66ffd640 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5602bc0bb920 0x5602bc0b7910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=86f75ab696edf23d server_cookie=46b9362d3844828 in_seq=0 out_seq=0 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.430+0000 7f4a46ffd640 10 client.15252 ms_handle_connect on v2:172.21.15.153:6834/3469968496 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a46ffd640 1 -- 172.21.15.106:0/12294168 <== mds.3 v2:172.21.15.153:6834/3469968496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5602bbfb24b0 con 0x5602bc0bb920 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a46ffd640 10 client.15252 handle_command_reply: tid=0 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 client.15252 shutdown 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 -- 172.21.15.106:0/12294168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 msgr2=0x5602bc055f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc055f70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c000c00 tx=0x7f4a5c005c90 comp rx=0 tx=0).stop 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 -- 172.21.15.106:0/12294168 shutdown_connections 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5602bc0bb920 0x5602bc0b7910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5602bc04b2a0 0x5602bc055f70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5602bc046130 0x5602bc052fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a46ffd640 0 client.15252 ms_handle_reset on v2:172.21.15.153:6834/3469968496 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 --2- 172.21.15.106:0/12294168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602bc045740 0x5602bc052a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 -- 172.21.15.106:0/12294168 >> 172.21.15.106:0/12294168 conn(0x5602bc00da60 msgr2=0x5602bc0476d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 -- 172.21.15.106:0/12294168 shutdown_connections 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 1 -- 172.21.15.106:0/12294168 wait complete. 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a70588740 20 client.15252 trim_cache size 0 max 16384 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 3, 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24446, 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:10:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 87.165905926999997, 2024-03-28T02:10:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:10:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 91, 2024-03-28T02:10:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:10:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 133.983406519 2024-03-28T02:10:36.432 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:10:36.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 -- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4a500a2800 msgr2=0x7f4a500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4a500a2800 0x7f4a500a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4a6009b800 tx=0x7f4a60086040 comp rx=0 tx=0).stop 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 -- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a68103260 msgr2=0x7f4a6819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a68103260 0x7f4a6819efc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000efd0 tx=0x7f4a6000c6a0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 -- 172.21.15.106:0/1272547175 shutdown_connections 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a68103260 0x7f4a6819efc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a681028a0 0x7f4a68197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f4a500a2800 0x7f4a500a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 --2- 172.21.15.106:0/1272547175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a681016b0 0x7f4a68197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.431+0000 7f4a6d99e640 1 -- 172.21.15.106:0/1272547175 >> 172.21.15.106:0/1272547175 conn(0x7f4a680fcc40 msgr2=0x7f4a68105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.432+0000 7f4a6d99e640 1 -- 172.21.15.106:0/1272547175 shutdown_connections 2024-03-28T02:10:36.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.432+0000 7f4a6d99e640 1 -- 172.21.15.106:0/1272547175 wait complete. 2024-03-28T02:10:36.435 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-28T02:10:36.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.546+0000 7fc1312f8640 1 -- 172.21.15.106:0/611823163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 msgr2=0x7fc12c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.546+0000 7fc1312f8640 1 --2- 172.21.15.106:0/611823163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 0x7fc12c101ad0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009900 tx=0x7fc11c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:36.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.546+0000 7fc1312f8640 1 -- 172.21.15.106:0/611823163 shutdown_connections 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.546+0000 7fc1312f8640 1 --2- 172.21.15.106:0/611823163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c103260 0x7fc12c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.546+0000 7fc1312f8640 1 --2- 172.21.15.106:0/611823163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc12c1028a0 0x7fc12c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.546+0000 7fc1312f8640 1 --2- 172.21.15.106:0/611823163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 0x7fc12c101ad0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.546+0000 7fc1312f8640 1 -- 172.21.15.106:0/611823163 >> 172.21.15.106:0/611823163 conn(0x7fc12c0fcc40 msgr2=0x7fc12c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.546+0000 7fc1312f8640 1 -- 172.21.15.106:0/611823163 shutdown_connections 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.546+0000 7fc1312f8640 1 -- 172.21.15.106:0/611823163 wait complete. 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc1312f8640 1 Processor -- start 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc1312f8640 1 -- start start 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc1312f8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 0x7fc12c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc1312f8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c1028a0 0x7fc12c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc1312f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc12c103260 0x7fc12c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc1312f8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c10d350 con 0x7fc12c1016b0 2024-03-28T02:10:36.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc1312f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c10d1d0 con 0x7fc12c103260 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc1312f8640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c10d4d0 con 0x7fc12c1028a0 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 0x7fc12c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 0x7fc12c197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56242/0 (socket says 172.21.15.106:56242) 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12affd640 1 -- 172.21.15.106:0/3064400587 learned_addr learned my addr 172.21.15.106:0/3064400587 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc123fff640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c1028a0 0x7fc12c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc12c103260 0x7fc12c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12affd640 1 -- 172.21.15.106:0/3064400587 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c1028a0 msgr2=0x7fc12c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12affd640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c1028a0 0x7fc12c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12affd640 1 -- 172.21.15.106:0/3064400587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc12c103260 msgr2=0x7fc12c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12affd640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc12c103260 0x7fc12c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12affd640 1 -- 172.21.15.106:0/3064400587 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c19f650 con 0x7fc12c1016b0 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc12affd640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 0x7fc12c197a30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc11c000c00 tx=0x7fc11c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.547+0000 7fc128ff9640 1 -- 172.21.15.106:0/3064400587 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c043020 con 0x7fc12c1016b0 2024-03-28T02:10:36.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.548+0000 7fc128ff9640 1 -- 172.21.15.106:0/3064400587 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc11c0054e0 con 0x7fc12c1016b0 2024-03-28T02:10:36.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.548+0000 7fc1312f8640 1 -- 172.21.15.106:0/3064400587 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c19f8e0 con 0x7fc12c1016b0 2024-03-28T02:10:36.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.548+0000 7fc128ff9640 1 -- 172.21.15.106:0/3064400587 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c003c40 con 0x7fc12c1016b0 2024-03-28T02:10:36.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.548+0000 7fc1312f8640 1 -- 172.21.15.106:0/3064400587 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c19fe30 con 0x7fc12c1016b0 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.550+0000 7fc128ff9640 1 -- 172.21.15.106:0/3064400587 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc11c046020 con 0x7fc12c1016b0 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.550+0000 7fc133fc0740 1 Processor -- start 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.550+0000 7fc133fc0740 1 -- start start 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d50ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d50fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 0x56443d512730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc133fc0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d46fc20 con 0x56443d50b130 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc133fc0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d418a40 con 0x56443d5102a0 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc133fc0740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d46bcc0 con 0x56443d50a740 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d50fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc128ff9640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc1180a2800 0x7fc1180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d50fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56252/0 (socket says 172.21.15.106:56252) 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12affd640 1 -- 172.21.15.106:0/4030319344 learned_addr learned my addr 172.21.15.106:0/4030319344 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d50ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc123fff640 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 0x56443d512730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc1180a2800 0x7fc1180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc128ff9640 1 -- 172.21.15.106:0/3064400587 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fc11c0f0e10 con 0x7fc12c1016b0 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12b7fe640 1 -- 172.21.15.106:0/4030319344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 msgr2=0x56443d512730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 0x56443d512730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12b7fe640 1 -- 172.21.15.106:0/4030319344 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 msgr2=0x56443d50fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d50fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12b7fe640 1 -- 172.21.15.106:0/4030319344 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56443d43af90 con 0x56443d50a740 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.551+0000 7fc12affd640 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d50fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d50ab60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc124005fd0 tx=0x7fc124012d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83da5d1ddac755d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc1180a2800 0x7fc1180a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc12401b000 tx=0x7fc12401f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc1217fa640 1 -- 172.21.15.106:0/4030319344 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124003970 con 0x56443d50a740 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc1217fa640 1 -- 172.21.15.106:0/4030319344 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc12401e760 con 0x56443d50a740 2024-03-28T02:10:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc1217fa640 1 -- 172.21.15.106:0/4030319344 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124004290 con 0x56443d50a740 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc133fc0740 1 -- 172.21.15.106:0/4030319344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 msgr2=0x56443d50ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc133fc0740 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d50ab60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc124005fd0 tx=0x7fc124012d80 comp rx=0 tx=0).stop 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc133fc0740 1 -- 172.21.15.106:0/4030319344 shutdown_connections 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc133fc0740 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 0x56443d512730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc133fc0740 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d50fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc133fc0740 1 --2- 172.21.15.106:0/4030319344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d50ab60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.552+0000 7fc133fc0740 1 -- 172.21.15.106:0/4030319344 >> 172.21.15.106:0/4030319344 conn(0x56443d4d2a60 msgr2=0x56443d5060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 -- 172.21.15.106:0/4030319344 shutdown_connections 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 -- 172.21.15.106:0/4030319344 wait complete. 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 -- start start 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 Processor -- start 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d514bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d515110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 0x56443d51c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d513190 con 0x56443d50b130 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d42be90 con 0x56443d5102a0 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d513310 con 0x56443d50a740 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc133fc0740 10 client.0 _ll_register_callbacks cb 0x56443d183ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc123fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d515110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc12affd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d514bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc12affd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d514bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:40338/0 (socket says 172.21.15.106:40338) 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc12affd640 1 -- 172.21.15.106:0/821004052 learned_addr learned my addr 172.21.15.106:0/821004052 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc123fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d515110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56260/0 (socket says 172.21.15.106:56260) 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 0x56443d51c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc12affd640 1 -- 172.21.15.106:0/821004052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 msgr2=0x56443d51c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc12affd640 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 0x56443d51c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc12affd640 1 -- 172.21.15.106:0/821004052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 msgr2=0x56443d515110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc12affd640 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d515110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.553+0000 7fc12affd640 1 -- 172.21.15.106:0/821004052 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56443d4389d0 con 0x56443d50a740 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc123fff640 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d515110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc12affd640 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d514bd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0ee620 tx=0x7fc11c02c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c1cce6fddf9680d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc1127fc640 10 client.0 ms_handle_connect on v2:172.21.15.178:3300/0 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc1127fc640 1 -- 172.21.15.106:0/821004052 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0bcdd0 con 0x56443d50a740 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc1127fc640 1 -- 172.21.15.106:0/821004052 <== mon.1 v2:172.21.15.178:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fc11c0bb770 con 0x56443d50a740 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc133fc0740 1 -- 172.21.15.106:0/821004052 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc12401e760 con 0x56443d50a740 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc1127fc640 1 -- 172.21.15.106:0/821004052 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc11c004250 con 0x56443d50a740 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc133fc0740 10 client.25036 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc133fc0740 1 -- 172.21.15.106:0/821004052 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56443d1842d0 con 0x56443d50a740 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc1127fc640 1 -- 172.21.15.106:0/821004052 <== mon.1 v2:172.21.15.178:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc12401e760 con 0x56443d50a740 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc1127fc640 1 -- 172.21.15.106:0/821004052 <== mon.1 v2:172.21.15.178:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fc11c0d6140 con 0x56443d50a740 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc133fc0740 10 client.25036 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc133fc0740 10 client.25036 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc133fc0740 20 client.25036 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc133fc0740 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x56443d521360 0x56443d5202b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc133fc0740 4 client.25036 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.554+0000 7fc133fc0740 1 -- 172.21.15.106:0/821004052 --> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56443d47d160 con 0x56443d521360 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.555+0000 7fc123fff640 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x56443d521360 0x56443d5202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.555+0000 7fc123fff640 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x56443d521360 0x56443d5202b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=531e617112a65985 server_cookie=afd87b1d1b962ef2 in_seq=0 out_seq=0 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.555+0000 7fc1127fc640 10 client.25036 ms_handle_connect on v2:172.21.15.178:6834/1402366472 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc1127fc640 1 -- 172.21.15.106:0/821004052 <== mds.2 v2:172.21.15.178:6834/1402366472 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56443d47d160 con 0x56443d521360 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc1127fc640 10 client.25036 handle_command_reply: tid=0 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 client.25036 shutdown 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 -- 172.21.15.106:0/821004052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 msgr2=0x56443d514bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d514bd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0ee620 tx=0x7fc11c02c880 comp rx=0 tx=0).stop 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 -- 172.21.15.106:0/821004052 shutdown_connections 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x56443d521360 0x56443d5202b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d5102a0 0x56443d51c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d50b130 0x56443d515110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc1127fc640 0 client.25036 ms_handle_reset on v2:172.21.15.178:6834/1402366472 2024-03-28T02:10:36.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 --2- 172.21.15.106:0/821004052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d514bd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 -- 172.21.15.106:0/821004052 >> 172.21.15.106:0/821004052 conn(0x56443d4d2a60 msgr2=0x56443d5052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 -- 172.21.15.106:0/821004052 shutdown_connections 2024-03-28T02:10:36.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 1 -- 172.21.15.106:0/821004052 wait complete. 2024-03-28T02:10:36.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.556+0000 7fc133fc0740 20 client.25036 trim_cache size 0 max 16384 2024-03-28T02:10:36.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc133fc0740 1 Processor -- start 2024-03-28T02:10:36.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc133fc0740 1 -- start start 2024-03-28T02:10:36.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50a740 0x56443d50bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50b130 0x56443d51c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d5147e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc133fc0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d520b80 con 0x56443d5102a0 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc133fc0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d5211e0 con 0x56443d50a740 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc133fc0740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d521840 con 0x56443d50b130 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d5147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d5147e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56276/0 (socket says 172.21.15.106:56276) 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc12b7fe640 1 -- 172.21.15.106:0/1055774903 learned_addr learned my addr 172.21.15.106:0/1055774903 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc12affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50a740 0x56443d50bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc123fff640 1 --2- 172.21.15.106:0/1055774903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50b130 0x56443d51c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc12b7fe640 1 -- 172.21.15.106:0/1055774903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50b130 msgr2=0x56443d51c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/1055774903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50b130 0x56443d51c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc12b7fe640 1 -- 172.21.15.106:0/1055774903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50a740 msgr2=0x56443d50bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/1055774903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50a740 0x56443d50bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc12b7fe640 1 -- 172.21.15.106:0/1055774903 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56443d24e6b0 con 0x56443d5102a0 2024-03-28T02:10:36.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.575+0000 7fc123fff640 1 --2- 172.21.15.106:0/1055774903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50b130 0x56443d51c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.576+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/1055774903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d5147e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc124011000 tx=0x7fc124009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7559c1f35a336b81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.576+0000 7fc120ff9640 1 -- 172.21.15.106:0/1055774903 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12402f070 con 0x56443d5102a0 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.576+0000 7fc120ff9640 1 -- 172.21.15.106:0/1055774903 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc124003e50 con 0x56443d5102a0 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.576+0000 7fc120ff9640 1 -- 172.21.15.106:0/1055774903 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124005300 con 0x56443d5102a0 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.576+0000 7fc133fc0740 1 -- 172.21.15.106:0/1055774903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 msgr2=0x56443d5147e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.576+0000 7fc133fc0740 1 --2- 172.21.15.106:0/1055774903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d5147e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc124011000 tx=0x7fc124009a10 comp rx=0 tx=0).stop 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 -- 172.21.15.106:0/1055774903 shutdown_connections 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 --2- 172.21.15.106:0/1055774903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d5147e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 --2- 172.21.15.106:0/1055774903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50b130 0x56443d51c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 --2- 172.21.15.106:0/1055774903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50a740 0x56443d50bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 -- 172.21.15.106:0/1055774903 >> 172.21.15.106:0/1055774903 conn(0x56443d4d2a60 msgr2=0x56443d50c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 -- 172.21.15.106:0/1055774903 shutdown_connections 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 -- 172.21.15.106:0/1055774903 wait complete. 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 -- start start 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 Processor -- start 2024-03-28T02:10:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d5182a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50b130 0x56443d5187e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d51b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d522500 con 0x56443d5102a0 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d521ea0 con 0x56443d50b130 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443d522b60 con 0x56443d50a740 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc133fc0740 10 client.0 _ll_register_callbacks cb 0x7fc11c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d51b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc123fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50b130 0x56443d5187e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc12affd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d5182a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d51b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56286/0 (socket says 172.21.15.106:56286) 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.577+0000 7fc12b7fe640 1 -- 172.21.15.106:0/505806077 learned_addr learned my addr 172.21.15.106:0/505806077 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.578+0000 7fc123fff640 1 -- 172.21.15.106:0/505806077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 msgr2=0x56443d5182a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.578+0000 7fc123fff640 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d5182a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.578+0000 7fc123fff640 1 -- 172.21.15.106:0/505806077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 msgr2=0x56443d51b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.578+0000 7fc123fff640 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d51b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.578+0000 7fc123fff640 1 -- 172.21.15.106:0/505806077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56443d51bdf0 con 0x56443d50b130 2024-03-28T02:10:36.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.578+0000 7fc12b7fe640 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d51b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:36.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.578+0000 7fc123fff640 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50b130 0x56443d5187e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x56443d522fb0 tx=0x7fc114022d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f76f1ed162ff016 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.578+0000 7fc1127fc640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc1127fc640 1 -- 172.21.15.106:0/505806077 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114012a60 con 0x56443d50b130 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc1127fc640 1 -- 172.21.15.106:0/505806077 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fc114012c00 con 0x56443d50b130 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc133fc0740 1 -- 172.21.15.106:0/505806077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc124003e50 con 0x56443d50b130 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc1127fc640 1 -- 172.21.15.106:0/505806077 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc114012e00 con 0x56443d50b130 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc133fc0740 10 client.25118 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc133fc0740 1 -- 172.21.15.106:0/505806077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56443d57ced0 con 0x56443d50b130 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc1127fc640 1 -- 172.21.15.106:0/505806077 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc124003e50 con 0x56443d50b130 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc1127fc640 1 -- 172.21.15.106:0/505806077 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fc114003c70 con 0x56443d50b130 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc133fc0740 10 client.25118 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc133fc0740 10 client.25118 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc133fc0740 20 client.25118 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc133fc0740 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x56443d581560 0x56443d57d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc133fc0740 4 client.25118 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc133fc0740 1 -- 172.21.15.106:0/505806077 --> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56443d4774b0 con 0x56443d581560 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.580+0000 7fc12affd640 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x56443d581560 0x56443d57d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc1127fc640 10 client.25118 ms_handle_connect on v2:172.21.15.178:6834/1402366472 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc12affd640 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x56443d581560 0x56443d57d1b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=141fafc4881ab298 server_cookie=eab03c4085ca2f03 in_seq=0 out_seq=0 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc1127fc640 1 -- 172.21.15.106:0/505806077 <== mds.2 v2:172.21.15.178:6834/1402366472 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x56443d4774b0 con 0x56443d581560 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc1127fc640 10 client.25118 handle_command_reply: tid=0 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 client.25118 shutdown 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 -- 172.21.15.106:0/505806077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50b130 msgr2=0x56443d5187e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50b130 0x56443d5187e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x56443d522fb0 tx=0x7fc114022d70 comp rx=0 tx=0).stop 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 -- 172.21.15.106:0/505806077 shutdown_connections 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x56443d581560 0x56443d57d1b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56443d5102a0 0x56443d51b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc1127fc640 0 client.25118 ms_handle_reset on v2:172.21.15.178:6834/1402366472 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56443d50b130 0x56443d5187e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 --2- 172.21.15.106:0/505806077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56443d50a740 0x56443d5182a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 -- 172.21.15.106:0/505806077 >> 172.21.15.106:0/505806077 conn(0x56443d4d2a60 msgr2=0x56443d5147c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 -- 172.21.15.106:0/505806077 shutdown_connections 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 1 -- 172.21.15.106:0/505806077 wait complete. 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.581+0000 7fc133fc0740 20 client.25118 trim_cache size 0 max 16384 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 2, 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24460, 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:10:36.579 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:10:36.580 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:10:36.580 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 88.333049493000004, 2024-03-28T02:10:36.580 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:10:36.580 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 91, 2024-03-28T02:10:36.580 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:10:36.580 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 131.06302930800001 2024-03-28T02:10:36.580 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 -- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc1180a2800 msgr2=0x7fc1180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc1180a2800 0x7fc1180a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc12401b000 tx=0x7fc12401f000 comp rx=0 tx=0).stop 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 -- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 msgr2=0x7fc12c197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 0x7fc12c197a30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc11c000c00 tx=0x7fc11c02c650 comp rx=0 tx=0).stop 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 -- 172.21.15.106:0/3064400587 shutdown_connections 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fc1180a2800 0x7fc1180a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc12c103260 0x7fc12c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c1028a0 0x7fc12c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 --2- 172.21.15.106:0/3064400587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc12c1016b0 0x7fc12c197a30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 -- 172.21.15.106:0/3064400587 >> 172.21.15.106:0/3064400587 conn(0x7fc12c0fcc40 msgr2=0x7fc12c105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 -- 172.21.15.106:0/3064400587 shutdown_connections 2024-03-28T02:10:36.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.582+0000 7fc1312f8640 1 -- 172.21.15.106:0/3064400587 wait complete. 2024-03-28T02:10:36.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T02:10:36.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.695+0000 7ff34320b640 1 -- 172.21.15.106:0/509765936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c102120 msgr2=0x7ff33c104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.695+0000 7ff34320b640 1 --2- 172.21.15.106:0/509765936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c102120 0x7ff33c104510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009900 tx=0x7ff32c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.695+0000 7ff34320b640 1 -- 172.21.15.106:0/509765936 shutdown_connections 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.695+0000 7ff34320b640 1 --2- 172.21.15.106:0/509765936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff33c107380 0x7ff33c1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.695+0000 7ff34320b640 1 --2- 172.21.15.106:0/509765936 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff33c104a50 0x7ff33c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.695+0000 7ff34320b640 1 --2- 172.21.15.106:0/509765936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c102120 0x7ff33c104510 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.695+0000 7ff34320b640 1 -- 172.21.15.106:0/509765936 >> 172.21.15.106:0/509765936 conn(0x7ff33c0fbb30 msgr2=0x7ff33c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.695+0000 7ff34320b640 1 -- 172.21.15.106:0/509765936 shutdown_connections 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.695+0000 7ff34320b640 1 -- 172.21.15.106:0/509765936 wait complete. 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff34320b640 1 Processor -- start 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff34320b640 1 -- start start 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff34320b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff33c102120 0x7ff33c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff34320b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c104a50 0x7ff33c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff34320b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff33c107380 0x7ff33c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff34320b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c10d330 con 0x7ff33c104a50 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff34320b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c10d1b0 con 0x7ff33c107380 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff34320b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c10d4b0 con 0x7ff33c102120 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff33bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c104a50 0x7ff33c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff33bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c104a50 0x7ff33c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56298/0 (socket says 172.21.15.106:56298) 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff33bfff640 1 -- 172.21.15.106:0/2203523164 learned_addr learned my addr 172.21.15.106:0/2203523164 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff340fa7640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff33c102120 0x7ff33c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff3417a8640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff33c107380 0x7ff33c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff33bfff640 1 -- 172.21.15.106:0/2203523164 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff33c102120 msgr2=0x7ff33c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff33c102120 0x7ff33c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff33bfff640 1 -- 172.21.15.106:0/2203523164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff33c107380 msgr2=0x7ff33c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff33c107380 0x7ff33c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.696+0000 7ff33bfff640 1 -- 172.21.15.106:0/2203523164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c19f650 con 0x7ff33c104a50 2024-03-28T02:10:36.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.697+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c104a50 0x7ff33c197f70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff33000e9f0 tx=0x7ff33000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.697+0000 7ff339ffb640 1 -- 172.21.15.106:0/2203523164 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33000ccc0 con 0x7ff33c104a50 2024-03-28T02:10:36.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.697+0000 7ff339ffb640 1 -- 172.21.15.106:0/2203523164 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff330005500 con 0x7ff33c104a50 2024-03-28T02:10:36.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.697+0000 7ff34320b640 1 -- 172.21.15.106:0/2203523164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33c19f940 con 0x7ff33c104a50 2024-03-28T02:10:36.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.697+0000 7ff339ffb640 1 -- 172.21.15.106:0/2203523164 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff330010690 con 0x7ff33c104a50 2024-03-28T02:10:36.693 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.697+0000 7ff34320b640 1 -- 172.21.15.106:0/2203523164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33c19fe90 con 0x7ff33c104a50 2024-03-28T02:10:36.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.699+0000 7ff339ffb640 1 -- 172.21.15.106:0/2203523164 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff3300046c0 con 0x7ff33c104a50 2024-03-28T02:10:36.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.699+0000 7ff339ffb640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff3240a2880 0x7ff3240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.699+0000 7ff345df3740 1 Processor -- start 2024-03-28T02:10:36.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.699+0000 7ff345df3740 1 -- start start 2024-03-28T02:10:36.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.699+0000 7ff340fa7640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff3240a2880 0x7ff3240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.699+0000 7ff345df3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc6af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff339ffb640 1 -- 172.21.15.106:0/2203523164 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7ff3300cf070 con 0x7ff33c104a50 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff345df3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc70040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff345df3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc72b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff345df3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbbcbc20 con 0x5578fbc6ab50 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff345df3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbb74a40 con 0x5578fbc6b540 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff345df3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbbc7cc0 con 0x5578fbc706b0 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff340fa7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc6af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff33bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc70040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff3417a8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc72b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff33bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc70040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:38162/0 (socket says 172.21.15.106:38162) 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff3417a8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc72b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:40384/0 (socket says 172.21.15.106:40384) 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff340fa7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc6af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56310/0 (socket says 172.21.15.106:56310) 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff33bfff640 1 -- 172.21.15.106:0/2860102457 learned_addr learned my addr 172.21.15.106:0/2860102457 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff340fa7640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff3240a2880 0x7ff3240a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff32c02c7b0 tx=0x7ff32c02cb90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff33bfff640 1 -- 172.21.15.106:0/2860102457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 msgr2=0x5578fbc72b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2860102457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc72b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff33bfff640 1 -- 172.21.15.106:0/2860102457 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 msgr2=0x5578fbc6af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2860102457 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc6af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff33bfff640 1 -- 172.21.15.106:0/2860102457 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578fbb96f90 con 0x5578fbc6b540 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff3417a8640 1 --2- 172.21.15.106:0/2860102457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc72b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2860102457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc70040 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff33009b660 tx=0x7ff33009b560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3637bcce31ec785 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff322ffd640 1 -- 172.21.15.106:0/2860102457 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3300b4070 con 0x5578fbc6b540 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff322ffd640 1 -- 172.21.15.106:0/2860102457 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff33009b6d0 con 0x5578fbc6b540 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.700+0000 7ff322ffd640 1 -- 172.21.15.106:0/2860102457 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff330097d40 con 0x5578fbc6b540 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 -- 172.21.15.106:0/2860102457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 msgr2=0x5578fbc70040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 --2- 172.21.15.106:0/2860102457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc70040 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff33009b660 tx=0x7ff33009b560 comp rx=0 tx=0).stop 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 -- 172.21.15.106:0/2860102457 shutdown_connections 2024-03-28T02:10:36.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 --2- 172.21.15.106:0/2860102457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc72b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 --2- 172.21.15.106:0/2860102457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc70040 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 --2- 172.21.15.106:0/2860102457 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc6af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 -- 172.21.15.106:0/2860102457 >> 172.21.15.106:0/2860102457 conn(0x5578fbc2ea60 msgr2=0x5578fbc620b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 -- 172.21.15.106:0/2860102457 shutdown_connections 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 -- 172.21.15.106:0/2860102457 wait complete. 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 -- start start 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.701+0000 7ff345df3740 1 Processor -- start 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff345df3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc75000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff345df3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc75540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff345df3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc7c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff345df3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbc73660 con 0x5578fbc6ab50 2024-03-28T02:10:36.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff345df3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbb87e90 con 0x5578fbc6b540 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff345df3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbc737e0 con 0x5578fbc706b0 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff345df3740 10 client.0 _ll_register_callbacks cb 0x5578fb8dfac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff340fa7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc75000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff340fa7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc75000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56314/0 (socket says 172.21.15.106:56314) 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff340fa7640 1 -- 172.21.15.106:0/570103655 learned_addr learned my addr 172.21.15.106:0/570103655 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff33bfff640 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc75540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff3417a8640 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc7c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff340fa7640 1 -- 172.21.15.106:0/570103655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 msgr2=0x5578fbc7c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff340fa7640 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc7c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff340fa7640 1 -- 172.21.15.106:0/570103655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 msgr2=0x5578fbc75540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff340fa7640 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc75540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff340fa7640 1 -- 172.21.15.106:0/570103655 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578fbb949d0 con 0x5578fbc6ab50 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff340fa7640 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc75000 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff32c02cfd0 tx=0x7ff32c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed922d66f9809a99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.702+0000 7ff31bfff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff31bfff640 1 -- 172.21.15.106:0/570103655 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c0048c0 con 0x5578fbc6ab50 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff31bfff640 1 -- 172.21.15.106:0/570103655 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7ff32c004a60 con 0x5578fbc6ab50 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff345df3740 1 -- 172.21.15.106:0/570103655 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff33009b6d0 con 0x5578fbc6ab50 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff31bfff640 1 -- 172.21.15.106:0/570103655 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff32c05a050 con 0x5578fbc6ab50 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff345df3740 10 client.15282 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff345df3740 1 -- 172.21.15.106:0/570103655 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5578fb8e02d0 con 0x5578fbc6ab50 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff31bfff640 1 -- 172.21.15.106:0/570103655 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff33009b6d0 con 0x5578fbc6ab50 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff31bfff640 1 -- 172.21.15.106:0/570103655 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff32c04c460 con 0x5578fbc6ab50 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff345df3740 10 client.15282 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff345df3740 10 client.15282 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff345df3740 20 client.15282 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff345df3740 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x5578fbc81810 0x5578fbc808c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff345df3740 4 client.15282 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff345df3740 1 -- 172.21.15.106:0/570103655 --> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578fbbd9160 con 0x5578fbc81810 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.703+0000 7ff33bfff640 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x5578fbc81810 0x5578fbc808c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.704+0000 7ff31bfff640 10 client.15282 ms_handle_connect on v2:172.21.15.178:6838/937216083 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.704+0000 7ff33bfff640 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x5578fbc81810 0x5578fbc808c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58da5c5fec65357d server_cookie=9b4092816ba508ba in_seq=0 out_seq=0 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.704+0000 7ff31bfff640 1 -- 172.21.15.106:0/570103655 <== mds.1 v2:172.21.15.178:6838/937216083 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5578fbbd9160 con 0x5578fbc81810 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.704+0000 7ff31bfff640 10 client.15282 handle_command_reply: tid=0 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.704+0000 7ff345df3740 1 client.15282 shutdown 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 -- 172.21.15.106:0/570103655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 msgr2=0x5578fbc75000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc75000 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff32c02cfd0 tx=0x7ff32c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 -- 172.21.15.106:0/570103655 shutdown_connections 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x5578fbc81810 0x5578fbc808c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc7c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc75540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff31bfff640 0 client.15282 ms_handle_reset on v2:172.21.15.178:6838/937216083 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 --2- 172.21.15.106:0/570103655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6ab50 0x5578fbc75000 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 -- 172.21.15.106:0/570103655 >> 172.21.15.106:0/570103655 conn(0x5578fbc2ea60 msgr2=0x5578fbc612f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 -- 172.21.15.106:0/570103655 shutdown_connections 2024-03-28T02:10:36.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 1 -- 172.21.15.106:0/570103655 wait complete. 2024-03-28T02:10:36.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.705+0000 7ff345df3740 20 client.15282 trim_cache size 0 max 16384 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff345df3740 1 Processor -- start 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff345df3740 1 -- start start 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff345df3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6ab50 0x5578fbc652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff345df3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6b540 0x5578fbc7c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff345df3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc65830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff345df3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbc81030 con 0x5578fbc6b540 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff345df3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbc81690 con 0x5578fbc6ab50 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff345df3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbc81cf0 con 0x5578fbc706b0 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff33bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6b540 0x5578fbc7c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff33bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6b540 0x5578fbc7c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56326/0 (socket says 172.21.15.106:56326) 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff33bfff640 1 -- 172.21.15.106:0/2454175360 learned_addr learned my addr 172.21.15.106:0/2454175360 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff3417a8640 1 --2- 172.21.15.106:0/2454175360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc65830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.721+0000 7ff340fa7640 1 --2- 172.21.15.106:0/2454175360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6ab50 0x5578fbc652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff33bfff640 1 -- 172.21.15.106:0/2454175360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 msgr2=0x5578fbc65830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2454175360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc65830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff33bfff640 1 -- 172.21.15.106:0/2454175360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6ab50 msgr2=0x5578fbc652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2454175360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6ab50 0x5578fbc652f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff33bfff640 1 -- 172.21.15.106:0/2454175360 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578fb9aa6b0 con 0x5578fbc6b540 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2454175360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6b540 0x5578fbc7c540 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3300ccba0 tx=0x7ff330004ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930426dc1f1831f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff3227fc640 1 -- 172.21.15.106:0/2454175360 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3300b4040 con 0x5578fbc6b540 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff3227fc640 1 -- 172.21.15.106:0/2454175360 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff33009b6d0 con 0x5578fbc6b540 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff3227fc640 1 -- 172.21.15.106:0/2454175360 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3300a39e0 con 0x5578fbc6b540 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff345df3740 1 -- 172.21.15.106:0/2454175360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6b540 msgr2=0x5578fbc7c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff345df3740 1 --2- 172.21.15.106:0/2454175360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6b540 0x5578fbc7c540 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3300ccba0 tx=0x7ff330004ba0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff345df3740 1 -- 172.21.15.106:0/2454175360 shutdown_connections 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff345df3740 1 --2- 172.21.15.106:0/2454175360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc706b0 0x5578fbc65830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff345df3740 1 --2- 172.21.15.106:0/2454175360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc6b540 0x5578fbc7c540 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff345df3740 1 --2- 172.21.15.106:0/2454175360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6ab50 0x5578fbc652f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.722+0000 7ff345df3740 1 -- 172.21.15.106:0/2454175360 >> 172.21.15.106:0/2454175360 conn(0x5578fbc2ea60 msgr2=0x5578fbc6ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 -- 172.21.15.106:0/2454175360 shutdown_connections 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 -- 172.21.15.106:0/2454175360 wait complete. 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 -- start start 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 Processor -- start 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc6ab50 0x5578fbc7be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc7c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc706b0 0x5578fbc788c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbc829b0 con 0x5578fbc706b0 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbc82350 con 0x5578fbc6b540 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578fbc83010 con 0x5578fbc6ab50 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff345df3740 10 client.0 _ll_register_callbacks cb 0x7ff32c004440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff3417a8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc706b0 0x5578fbc788c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff340fa7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc6ab50 0x5578fbc7be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff3417a8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc706b0 0x5578fbc788c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56340/0 (socket says 172.21.15.106:56340) 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff340fa7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc6ab50 0x5578fbc7be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:40414/0 (socket says 172.21.15.106:40414) 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff3417a8640 1 -- 172.21.15.106:0/2849011604 learned_addr learned my addr 172.21.15.106:0/2849011604 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.723+0000 7ff33bfff640 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc7c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff3417a8640 1 -- 172.21.15.106:0/2849011604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc6ab50 msgr2=0x5578fbc7be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff3417a8640 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc6ab50 0x5578fbc7be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff3417a8640 1 -- 172.21.15.106:0/2849011604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 msgr2=0x5578fbc7c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff3417a8640 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc7c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff3417a8640 1 -- 172.21.15.106:0/2849011604 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578fbc78f80 con 0x5578fbc706b0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff3417a8640 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc706b0 0x5578fbc788c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff334012d50 tx=0x7ff33400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c077bbfce82748 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff31bfff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff31bfff640 1 -- 172.21.15.106:0/2849011604 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334022060 con 0x5578fbc706b0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff31bfff640 1 -- 172.21.15.106:0/2849011604 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7ff334003b70 con 0x5578fbc706b0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff345df3740 1 -- 172.21.15.106:0/2849011604 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff33009b6d0 con 0x5578fbc706b0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff31bfff640 1 -- 172.21.15.106:0/2849011604 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff334029050 con 0x5578fbc706b0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff345df3740 10 client.15288 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff345df3740 1 -- 172.21.15.106:0/2849011604 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5578fbcdd540 con 0x5578fbc706b0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff31bfff640 1 -- 172.21.15.106:0/2849011604 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff33009b6d0 con 0x5578fbc706b0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff31bfff640 1 -- 172.21.15.106:0/2849011604 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff334010430 con 0x5578fbc706b0 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff345df3740 10 client.15288 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff345df3740 10 client.15288 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff345df3740 20 client.15288 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff345df3740 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x5578fbce1c50 0x5578fbcdd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff345df3740 4 client.15288 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.724+0000 7ff345df3740 1 -- 172.21.15.106:0/2849011604 --> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5578fbbd34b0 con 0x5578fbce1c50 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff340fa7640 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x5578fbce1c50 0x5578fbcdd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff31bfff640 10 client.15288 ms_handle_connect on v2:172.21.15.178:6838/937216083 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff340fa7640 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x5578fbce1c50 0x5578fbcdd880 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75f7e0a2719bd7b9 server_cookie=ce8fc49247292e38 in_seq=0 out_seq=0 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff31bfff640 1 -- 172.21.15.106:0/2849011604 <== mds.1 v2:172.21.15.178:6838/937216083 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5578fbbd34b0 con 0x5578fbce1c50 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff31bfff640 10 client.15288 handle_command_reply: tid=0 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 client.15288 shutdown 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 -- 172.21.15.106:0/2849011604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc706b0 msgr2=0x5578fbc788c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc706b0 0x5578fbc788c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff334012d50 tx=0x7ff33400f790 comp rx=0 tx=0).stop 2024-03-28T02:10:36.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 -- 172.21.15.106:0/2849011604 shutdown_connections 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x5578fbce1c50 0x5578fbcdd880 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578fbc706b0 0x5578fbc788c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff31bfff640 0 client.15288 ms_handle_reset on v2:172.21.15.178:6838/937216083 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5578fbc6b540 0x5578fbc7c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 --2- 172.21.15.106:0/2849011604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5578fbc6ab50 0x5578fbc7be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 -- 172.21.15.106:0/2849011604 >> 172.21.15.106:0/2849011604 conn(0x5578fbc2ea60 msgr2=0x5578fbc652f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 -- 172.21.15.106:0/2849011604 shutdown_connections 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 1 -- 172.21.15.106:0/2849011604 wait complete. 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.725+0000 7ff345df3740 20 client.15288 trim_cache size 0 max 16384 2024-03-28T02:10:36.726 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 1, 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24499, 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 89.202928184000001, 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 91, 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 122.68244131500001 2024-03-28T02:10:36.727 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:10:36.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 -- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff3240a2880 msgr2=0x7ff3240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff3240a2880 0x7ff3240a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff32c02c7b0 tx=0x7ff32c02cb90 comp rx=0 tx=0).stop 2024-03-28T02:10:36.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 -- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c104a50 msgr2=0x7ff33c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c104a50 0x7ff33c197f70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff33000e9f0 tx=0x7ff33000eeb0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 -- 172.21.15.106:0/2203523164 shutdown_connections 2024-03-28T02:10:36.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff3240a2880 0x7ff3240a4d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff33c107380 0x7ff33c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff33c104a50 0x7ff33c197f70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 --2- 172.21.15.106:0/2203523164 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff33c102120 0x7ff33c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 -- 172.21.15.106:0/2203523164 >> 172.21.15.106:0/2203523164 conn(0x7ff33c0fbb30 msgr2=0x7ff33c102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 -- 172.21.15.106:0/2203523164 shutdown_connections 2024-03-28T02:10:36.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.726+0000 7ff34320b640 1 -- 172.21.15.106:0/2203523164 wait complete. 2024-03-28T02:10:36.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:36 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1069005852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:36.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:36 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4186478894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:10:36.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T02:10:36.835 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 -- 172.21.15.106:0/2821268134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76a41005c0 msgr2=0x7f76a41009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.835 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/2821268134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76a41005c0 0x7f76a41009e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7694009900 tx=0x7f769402bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:36.835 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 -- 172.21.15.106:0/2821268134 shutdown_connections 2024-03-28T02:10:36.835 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/2821268134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a4102170 0x7f76a4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.835 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/2821268134 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76a41017b0 0x7f76a4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/2821268134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76a41005c0 0x7f76a41009e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 -- 172.21.15.106:0/2821268134 >> 172.21.15.106:0/2821268134 conn(0x7f76a40fbb30 msgr2=0x7f76a40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 -- 172.21.15.106:0/2821268134 shutdown_connections 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 -- 172.21.15.106:0/2821268134 wait complete. 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 Processor -- start 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 -- start start 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76a41005c0 0x7f76a4196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76a41017b0 0x7f76a4196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a4102170 0x7f76a419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a410c230 con 0x7f76a41005c0 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a410c0b0 con 0x7f76a4102170 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.840+0000 7f76ac6ac640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a410c3b0 con 0x7f76a41017b0 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aa448640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76a41005c0 0x7f76a4196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aa448640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76a41005c0 0x7f76a4196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56364/0 (socket says 172.21.15.106:56364) 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aa448640 1 -- 172.21.15.106:0/1741986480 learned_addr learned my addr 172.21.15.106:0/1741986480 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76a9c47640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76a41017b0 0x7f76a4196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aac49640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a4102170 0x7f76a419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aa448640 1 -- 172.21.15.106:0/1741986480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76a41017b0 msgr2=0x7f76a4196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aa448640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76a41017b0 0x7f76a4196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aa448640 1 -- 172.21.15.106:0/1741986480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a4102170 msgr2=0x7f76a419df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aa448640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a4102170 0x7f76a419df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aa448640 1 -- 172.21.15.106:0/1741986480 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a419e5a0 con 0x7f76a41005c0 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76aa448640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76a41005c0 0x7f76a4196980 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7694000c00 tx=0x7f7694004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f769b7fe640 1 -- 172.21.15.106:0/1741986480 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694004450 con 0x7f76a41005c0 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76ac6ac640 1 -- 172.21.15.106:0/1741986480 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a4069940 con 0x7f76a41005c0 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f769b7fe640 1 -- 172.21.15.106:0/1741986480 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76940054e0 con 0x7f76a41005c0 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f769b7fe640 1 -- 172.21.15.106:0/1741986480 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769403b440 con 0x7f76a41005c0 2024-03-28T02:10:36.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.841+0000 7f76ac6ac640 1 -- 172.21.15.106:0/1741986480 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a4069e80 con 0x7f76a41005c0 2024-03-28T02:10:36.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.843+0000 7f769b7fe640 1 -- 172.21.15.106:0/1741986480 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7694003710 con 0x7f76a41005c0 2024-03-28T02:10:36.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76af373740 1 Processor -- start 2024-03-28T02:10:36.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76af373740 1 -- start start 2024-03-28T02:10:36.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f769b7fe640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f76900a2800 0x7f76900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76af373740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def018b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76af373740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def019130 0x564def01dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76af373740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def01e2a0 0x564def020730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76af373740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x564deef7dc20 con 0x564def018740 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76af373740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x564deef26a40 con 0x564def019130 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76af373740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564deef79cc0 con 0x564def01e2a0 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76aac49640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f76900a2800 0x7f76900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76aa448640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def018b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76aac49640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def01e2a0 0x564def020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76a9c47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def019130 0x564def01dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76aa448640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def018b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56378/0 (socket says 172.21.15.106:56378) 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f76aa448640 1 -- 172.21.15.106:0/277830535 learned_addr learned my addr 172.21.15.106:0/277830535 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.844+0000 7f769b7fe640 1 -- 172.21.15.106:0/1741986480 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f76940f27f0 con 0x7f76a41005c0 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f76aa448640 1 -- 172.21.15.106:0/277830535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def01e2a0 msgr2=0x564def020730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f76aa448640 1 --2- 172.21.15.106:0/277830535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def01e2a0 0x564def020730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f76aa448640 1 -- 172.21.15.106:0/277830535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def019130 msgr2=0x564def01dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f76aa448640 1 --2- 172.21.15.106:0/277830535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def019130 0x564def01dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f76aa448640 1 -- 172.21.15.106:0/277830535 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564deef48f90 con 0x564def018740 2024-03-28T02:10:36.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f76a9c47640 1 --2- 172.21.15.106:0/277830535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def019130 0x564def01dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f76aac49640 1 --2- 172.21.15.106:0/277830535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def01e2a0 0x564def020730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f76aac49640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f76900a2800 0x7f76900a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f769c005fd0 tx=0x7f769c009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f76aa448640 1 --2- 172.21.15.106:0/277830535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def018b60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f76940effb0 tx=0x7f7694004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fbb9684aa78e1a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f7698ff9640 1 -- 172.21.15.106:0/277830535 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76940be7b0 con 0x564def018740 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f7698ff9640 1 -- 172.21.15.106:0/277830535 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7694004c80 con 0x564def018740 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.845+0000 7f7698ff9640 1 -- 172.21.15.106:0/277830535 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76940d7f10 con 0x564def018740 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 -- 172.21.15.106:0/277830535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 msgr2=0x564def018b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 --2- 172.21.15.106:0/277830535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def018b60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f76940effb0 tx=0x7f7694004bd0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 -- 172.21.15.106:0/277830535 shutdown_connections 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 --2- 172.21.15.106:0/277830535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def01e2a0 0x564def020730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 --2- 172.21.15.106:0/277830535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def019130 0x564def01dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 --2- 172.21.15.106:0/277830535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def018b60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 -- 172.21.15.106:0/277830535 >> 172.21.15.106:0/277830535 conn(0x564deefe0a60 msgr2=0x564def0140b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 -- 172.21.15.106:0/277830535 shutdown_connections 2024-03-28T02:10:36.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 -- 172.21.15.106:0/277830535 wait complete. 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 -- start start 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 Processor -- start 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def0227a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def022ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def01e2a0 0x564def029d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x564deef7dc20 con 0x564def018740 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x564deef39e90 con 0x564def01e2a0 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564def021010 con 0x564def019130 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76af373740 10 client.0 _ll_register_callbacks cb 0x564deec91ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76aa448640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def0227a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76aa448640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def0227a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56380/0 (socket says 172.21.15.106:56380) 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76aa448640 1 -- 172.21.15.106:0/867780030 learned_addr learned my addr 172.21.15.106:0/867780030 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.846+0000 7f76a9c47640 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def022ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76aac49640 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def01e2a0 0x564def029d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76aa448640 1 -- 172.21.15.106:0/867780030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 msgr2=0x564def022ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76aa448640 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def022ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76aa448640 1 -- 172.21.15.106:0/867780030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def01e2a0 msgr2=0x564def029d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76aa448640 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def01e2a0 0x564def029d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76aa448640 1 -- 172.21.15.106:0/867780030 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564deef469d0 con 0x564def018740 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76aa448640 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def0227a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f76940c70c0 tx=0x7f76940be9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c7e009a76fd21 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f768dffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f768dffb640 1 -- 172.21.15.106:0/867780030 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76940d80f0 con 0x564def018740 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f768dffb640 1 -- 172.21.15.106:0/867780030 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f76940bf2b0 con 0x564def018740 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76af373740 1 -- 172.21.15.106:0/867780030 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7694004c80 con 0x564def018740 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f768dffb640 1 -- 172.21.15.106:0/867780030 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7694005690 con 0x564def018740 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76af373740 10 client.15300 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76af373740 1 -- 172.21.15.106:0/867780030 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564deec922d0 con 0x564def018740 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f768dffb640 1 -- 172.21.15.106:0/867780030 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7694004c80 con 0x564def018740 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f768dffb640 1 -- 172.21.15.106:0/867780030 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f76940da070 con 0x564def018740 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76af373740 10 client.15300 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76af373740 10 client.15300 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76af373740 20 client.15300 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.847+0000 7f76af373740 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x564def02ec10 0x564def02de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.848+0000 7f76af373740 4 client.15300 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.848+0000 7f76af373740 1 -- 172.21.15.106:0/867780030 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564deef8b160 con 0x564def02ec10 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.848+0000 7f76a9c47640 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x564def02ec10 0x564def02de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.848+0000 7f76a9c47640 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x564def02ec10 0x564def02de80 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a10d13c83efc38f server_cookie=f06d4984ec3a1833 in_seq=0 out_seq=0 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.848+0000 7f768dffb640 10 client.15300 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f768dffb640 1 -- 172.21.15.106:0/867780030 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564deef8b160 con 0x564def02ec10 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f768dffb640 10 client.15300 handle_command_reply: tid=0 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 client.15300 shutdown 2024-03-28T02:10:36.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 -- 172.21.15.106:0/867780030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 msgr2=0x564def0227a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def0227a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f76940c70c0 tx=0x7f76940be9f0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 -- 172.21.15.106:0/867780030 shutdown_connections 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x564def02ec10 0x564def02de80 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def01e2a0 0x564def029d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def022ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 --2- 172.21.15.106:0/867780030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def0227a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f768dffb640 0 client.15300 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 -- 172.21.15.106:0/867780030 >> 172.21.15.106:0/867780030 conn(0x564deefe0a60 msgr2=0x564def0132f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 -- 172.21.15.106:0/867780030 shutdown_connections 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 1 -- 172.21.15.106:0/867780030 wait complete. 2024-03-28T02:10:36.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.849+0000 7f76af373740 20 client.15300 trim_cache size 0 max 16384 2024-03-28T02:10:36.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76af373740 1 Processor -- start 2024-03-28T02:10:36.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76af373740 1 -- start start 2024-03-28T02:10:36.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76af373740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564deefe0a60 0x564def022c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76af373740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def029cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76af373740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def017a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76af373740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x564def02e430 con 0x564def018740 2024-03-28T02:10:36.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76af373740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x564def02ea90 con 0x564deefe0a60 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76af373740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564def02f0f0 con 0x564def019130 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76a9c47640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def029cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76a9c47640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def029cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56384/0 (socket says 172.21.15.106:56384) 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76a9c47640 1 -- 172.21.15.106:0/3718973735 learned_addr learned my addr 172.21.15.106:0/3718973735 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76aa448640 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564deefe0a60 0x564def022c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76aac49640 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def017a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76a9c47640 1 -- 172.21.15.106:0/3718973735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 msgr2=0x564def017a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76a9c47640 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def017a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76a9c47640 1 -- 172.21.15.106:0/3718973735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564deefe0a60 msgr2=0x564def022c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76a9c47640 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564deefe0a60 0x564def022c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76a9c47640 1 -- 172.21.15.106:0/3718973735 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564deed5c6b0 con 0x564def018740 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76aac49640 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def017a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f76a9c47640 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def029cb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f76a0009e50 tx=0x7f76a0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0156f451e615d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f768ffff640 1 -- 172.21.15.106:0/3718973735 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a0024020 con 0x564def018740 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f768ffff640 1 -- 172.21.15.106:0/3718973735 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76a00200f0 con 0x564def018740 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.866+0000 7f768ffff640 1 -- 172.21.15.106:0/3718973735 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a000f7f0 con 0x564def018740 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 -- 172.21.15.106:0/3718973735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 msgr2=0x564def029cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def029cb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f76a0009e50 tx=0x7f76a0005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 -- 172.21.15.106:0/3718973735 shutdown_connections 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def017a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564def018740 0x564def029cb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 --2- 172.21.15.106:0/3718973735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564deefe0a60 0x564def022c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 -- 172.21.15.106:0/3718973735 >> 172.21.15.106:0/3718973735 conn(0x564def021670 msgr2=0x564def0172f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 -- 172.21.15.106:0/3718973735 shutdown_connections 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 -- 172.21.15.106:0/3718973735 wait complete. 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 -- start start 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.867+0000 7f76af373740 1 Processor -- start 2024-03-28T02:10:36.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564deefe0a60 0x564def029c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def018740 0x564def026720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def026d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x564def02fdb0 con 0x564deefe0a60 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x564def02f750 con 0x564def018740 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564def030410 con 0x564def019130 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 10 client.0 _ll_register_callbacks cb 0x7f76940db7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aa448640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564deefe0a60 0x564def029c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aa448640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564deefe0a60 0x564def029c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:56398/0 (socket says 172.21.15.106:56398) 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aa448640 1 -- 172.21.15.106:0/3631537899 learned_addr learned my addr 172.21.15.106:0/3631537899 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76a9c47640 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def018740 0x564def026720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aac49640 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def026d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aa448640 1 -- 172.21.15.106:0/3631537899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 msgr2=0x564def026d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aa448640 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def026d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aa448640 1 -- 172.21.15.106:0/3631537899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def018740 msgr2=0x564def026720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aa448640 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def018740 0x564def026720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aa448640 1 -- 172.21.15.106:0/3631537899 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564def089cf0 con 0x564deefe0a60 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76aa448640 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564deefe0a60 0x564def029c80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f76940db7c0 tx=0x7f76940be870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd36d3b060429de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f768dffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f768dffb640 1 -- 172.21.15.106:0/3631537899 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76940bb0f0 con 0x564deefe0a60 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f768dffb640 1 -- 172.21.15.106:0/3631537899 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f76940bf0b0 con 0x564deefe0a60 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 1 -- 172.21.15.106:0/3631537899 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76a00200f0 con 0x564deefe0a60 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f768dffb640 1 -- 172.21.15.106:0/3631537899 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7694004c80 con 0x564deefe0a60 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 10 client.15306 fetch_fsmap learned FSMap version 28 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.868+0000 7f76af373740 1 -- 172.21.15.106:0/3631537899 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564def089cf0 con 0x564deefe0a60 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f768dffb640 1 -- 172.21.15.106:0/3631537899 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76a00200f0 con 0x564deefe0a60 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f768dffb640 1 -- 172.21.15.106:0/3631537899 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f76940da030 con 0x564deefe0a60 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f76af373740 10 client.15306 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:10:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f76af373740 10 client.15306 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f76af373740 20 client.15306 populate_metadata read hostname 'smithi106' 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f76af373740 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x564def01e2a0 0x564def08b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f76af373740 4 client.15306 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f76af373740 1 -- 172.21.15.106:0/3631537899 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564deef854b0 con 0x564def01e2a0 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f76a9c47640 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x564def01e2a0 0x564def08b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f768dffb640 10 client.15306 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.869+0000 7f76a9c47640 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x564def01e2a0 0x564def08b0b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=265eaed78ad77b73 server_cookie=b1b8dc73f841d905 in_seq=0 out_seq=0 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f768dffb640 1 -- 172.21.15.106:0/3631537899 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x564deef854b0 con 0x564def01e2a0 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f768dffb640 10 client.15306 handle_command_reply: tid=0 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 client.15306 shutdown 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 -- 172.21.15.106:0/3631537899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564deefe0a60 msgr2=0x564def029c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564deefe0a60 0x564def029c80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f76940db7c0 tx=0x7f76940be870 comp rx=0 tx=0).stop 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 -- 172.21.15.106:0/3631537899 shutdown_connections 2024-03-28T02:10:36.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x564def01e2a0 0x564def08b0b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564def019130 0x564def026d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f768dffb640 0 client.15306 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564def018740 0x564def026720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 --2- 172.21.15.106:0/3631537899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564deefe0a60 0x564def029c80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 -- 172.21.15.106:0/3631537899 >> 172.21.15.106:0/3631537899 conn(0x564def021670 msgr2=0x564def022a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 -- 172.21.15.106:0/3631537899 shutdown_connections 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 1 -- 172.21.15.106:0/3631537899 wait complete. 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76af373740 20 client.15306 trim_cache size 0 max 16384 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:10:36.870 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 0, 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24509, 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 107.42500756699999, 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 91, 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 114.11990492 2024-03-28T02:10:36.871 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:10:36.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 -- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f76900a2800 msgr2=0x7f76900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f76900a2800 0x7f76900a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f769c005fd0 tx=0x7f769c009b80 comp rx=0 tx=0).stop 2024-03-28T02:10:36.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 -- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76a41005c0 msgr2=0x7f76a4196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:36.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76a41005c0 0x7f76a4196980 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7694000c00 tx=0x7f7694004960 comp rx=0 tx=0).stop 2024-03-28T02:10:36.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 -- 172.21.15.106:0/1741986480 shutdown_connections 2024-03-28T02:10:36.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f76900a2800 0x7f76900a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a4102170 0x7f76a419df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76a41017b0 0x7f76a4196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 --2- 172.21.15.106:0/1741986480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76a41005c0 0x7f76a4196980 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:36.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 -- 172.21.15.106:0/1741986480 >> 172.21.15.106:0/1741986480 conn(0x7f76a40fbb30 msgr2=0x7f76a4104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:36.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 -- 172.21.15.106:0/1741986480 shutdown_connections 2024-03-28T02:10:36.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:36.870+0000 7f76ac6ac640 1 -- 172.21.15.106:0/1741986480 wait complete. 2024-03-28T02:10:36.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:36 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1069005852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:36.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:36 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4186478894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:10:36.874 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T02:10:36.874 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T02:10:36.884 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:10:36.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T02:10:36.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1069005852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:36.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:36 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4186478894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:10:37.123 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:37.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:37 smithi106 ceph-mon[31965]: pgmap v348: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:37 smithi178 ceph-mon[36521]: pgmap v348: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:37.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:37 smithi153 ceph-mon[36228]: pgmap v348: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:38.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.927+0000 7fbd96b51700 1 -- 172.21.15.106:0/2642974748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 msgr2=0x7fbd90102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.927+0000 7fbd96b51700 1 --2- 172.21.15.106:0/2642974748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 0x7fbd90102720 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbd840099c0 tx=0x7fbd84009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.927+0000 7fbd96b51700 1 -- 172.21.15.106:0/2642974748 shutdown_connections 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.927+0000 7fbd96b51700 1 --2- 172.21.15.106:0/2642974748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd90102c60 0x7fbd90109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.927+0000 7fbd96b51700 1 --2- 172.21.15.106:0/2642974748 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd901010c0 0x7fbd901014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.927+0000 7fbd96b51700 1 --2- 172.21.15.106:0/2642974748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 0x7fbd90102720 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.927+0000 7fbd96b51700 1 -- 172.21.15.106:0/2642974748 >> 172.21.15.106:0/2642974748 conn(0x7fbd900fc680 msgr2=0x7fbd900feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.927+0000 7fbd96b51700 1 -- 172.21.15.106:0/2642974748 shutdown_connections 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.927+0000 7fbd96b51700 1 -- 172.21.15.106:0/2642974748 wait complete. 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd96b51700 1 Processor -- start 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd96b51700 1 -- start start 2024-03-28T02:10:38.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd96b51700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd901010c0 0x7fbd901972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd96b51700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 0x7fbd90197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd96b51700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd90102c60 0x7fbd9019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd96b51700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9010cca0 con 0x7fbd901022c0 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd96b51700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9010cb20 con 0x7fbd901010c0 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd96b51700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9010ce20 con 0x7fbd90102c60 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd8ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 0x7fbd90197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd8ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 0x7fbd90197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54520/0 (socket says 172.21.15.106:54520) 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd8ffff700 1 -- 172.21.15.106:0/1674921764 learned_addr learned my addr 172.21.15.106:0/1674921764 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd948ed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd901010c0 0x7fbd901972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.928+0000 7fbd950ee700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd90102c60 0x7fbd9019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd8ffff700 1 -- 172.21.15.106:0/1674921764 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd90102c60 msgr2=0x7fbd9019e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd8ffff700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd90102c60 0x7fbd9019e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd8ffff700 1 -- 172.21.15.106:0/1674921764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd901010c0 msgr2=0x7fbd901972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd8ffff700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd901010c0 0x7fbd901972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:38.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd8ffff700 1 -- 172.21.15.106:0/1674921764 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9019ef40 con 0x7fbd901022c0 2024-03-28T02:10:38.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd8ffff700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 0x7fbd90197820 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbd84005910 tx=0x7fbd8400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:38.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd8dffb700 1 -- 172.21.15.106:0/1674921764 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84009e60 con 0x7fbd901022c0 2024-03-28T02:10:38.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd8dffb700 1 -- 172.21.15.106:0/1674921764 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd84003960 con 0x7fbd901022c0 2024-03-28T02:10:38.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9019f1d0 con 0x7fbd901022c0 2024-03-28T02:10:38.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd8dffb700 1 -- 172.21.15.106:0/1674921764 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840173f0 con 0x7fbd901022c0 2024-03-28T02:10:38.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.929+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9019f680 con 0x7fbd901022c0 2024-03-28T02:10:38.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.930+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd90068e50 con 0x7fbd901022c0 2024-03-28T02:10:38.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.931+0000 7fbd8dffb700 1 -- 172.21.15.106:0/1674921764 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbd84003b10 con 0x7fbd901022c0 2024-03-28T02:10:38.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.932+0000 7fbd8dffb700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd780a2910 0x7fbd780a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:38.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.932+0000 7fbd8dffb700 1 -- 172.21.15.106:0/1674921764 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fbd840cfad0 con 0x7fbd901022c0 2024-03-28T02:10:38.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.932+0000 7fbd948ed700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd780a2910 0x7fbd780a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:38.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.932+0000 7fbd948ed700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd780a2910 0x7fbd780a4dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbd90102120 tx=0x7fbd8800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:38.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:38.936+0000 7fbd8dffb700 1 -- 172.21.15.106:0/1674921764 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd84013070 con 0x7fbd901022c0 2024-03-28T02:10:39.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.151+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd900625d0 con 0x7fbd780a2910 2024-03-28T02:10:39.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.152+0000 7fbd8dffb700 1 -- 172.21.15.106:0/1674921764 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fbd900625d0 con 0x7fbd780a2910 2024-03-28T02:10:39.144 INFO:teuthology.orchestra.run.smithi106.stdout:Backend: cephadm 2024-03-28T02:10:39.144 INFO:teuthology.orchestra.run.smithi106.stdout:Available: Yes 2024-03-28T02:10:39.144 INFO:teuthology.orchestra.run.smithi106.stdout:Paused: No 2024-03-28T02:10:39.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd780a2910 msgr2=0x7fbd780a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:39.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd780a2910 0x7fbd780a4dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbd90102120 tx=0x7fbd8800b040 comp rx=0 tx=0).stop 2024-03-28T02:10:39.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 msgr2=0x7fbd90197820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:39.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 0x7fbd90197820 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbd84005910 tx=0x7fbd8400f690 comp rx=0 tx=0).stop 2024-03-28T02:10:39.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 shutdown_connections 2024-03-28T02:10:39.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd90102c60 0x7fbd9019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:39.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd901010c0 0x7fbd901972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:39.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd780a2910 0x7fbd780a4dd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:39.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 --2- 172.21.15.106:0/1674921764 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd901022c0 0x7fbd90197820 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:39.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 >> 172.21.15.106:0/1674921764 conn(0x7fbd900fc680 msgr2=0x7fbd901055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:39.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 shutdown_connections 2024-03-28T02:10:39.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:39.154+0000 7fbd96b51700 1 -- 172.21.15.106:0/1674921764 wait complete. 2024-03-28T02:10:39.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:39 smithi106 ceph-mon[31965]: pgmap v349: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:10:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:39 smithi178 ceph-mon[36521]: pgmap v349: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:10:39.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T02:10:39.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:39 smithi153 ceph-mon[36228]: pgmap v349: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:10:40.204 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:40.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:40 smithi106 ceph-mon[31965]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:40.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:40 smithi178 ceph-mon[36521]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:40.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:40 smithi153 ceph-mon[36228]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:41.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:41 smithi106 ceph-mon[31965]: pgmap v350: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:41 smithi178 ceph-mon[36521]: pgmap v350: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:41.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:41 smithi153 ceph-mon[36228]: pgmap v350: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T02:10:42.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.080+0000 7f7d8f592700 1 -- 172.21.15.106:0/2669371419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 msgr2=0x7f7d881014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:42.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.080+0000 7f7d8f592700 1 --2- 172.21.15.106:0/2669371419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 0x7f7d881014e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7d780099c0 tx=0x7f7d78009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.080+0000 7f7d8f592700 1 -- 172.21.15.106:0/2669371419 shutdown_connections 2024-03-28T02:10:42.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.080+0000 7f7d8f592700 1 --2- 172.21.15.106:0/2669371419 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d88102c60 0x7f7d88109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.080+0000 7f7d8f592700 1 --2- 172.21.15.106:0/2669371419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 0x7f7d881014e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.080+0000 7f7d8f592700 1 --2- 172.21.15.106:0/2669371419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d881022c0 0x7f7d88102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.080+0000 7f7d8f592700 1 -- 172.21.15.106:0/2669371419 >> 172.21.15.106:0/2669371419 conn(0x7f7d880fc680 msgr2=0x7f7d880feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:42.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.080+0000 7f7d8f592700 1 -- 172.21.15.106:0/2669371419 shutdown_connections 2024-03-28T02:10:42.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.080+0000 7f7d8f592700 1 -- 172.21.15.106:0/2669371419 wait complete. 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8f592700 1 Processor -- start 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8f592700 1 -- start start 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8f592700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 0x7f7d88197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8f592700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d881022c0 0x7f7d88197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8f592700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d88102c60 0x7f7d8819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8f592700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8810cc90 con 0x7f7d881010c0 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8f592700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8810cb10 con 0x7f7d881022c0 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8f592700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8810ce10 con 0x7f7d88102c60 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8d32e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 0x7f7d88197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8d32e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 0x7f7d88197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54530/0 (socket says 172.21.15.106:54530) 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8d32e700 1 -- 172.21.15.106:0/1814391918 learned_addr learned my addr 172.21.15.106:0/1814391918 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8db2f700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d88102c60 0x7f7d8819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8cb2d700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d881022c0 0x7f7d88197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:42.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8d32e700 1 -- 172.21.15.106:0/1814391918 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d88102c60 msgr2=0x7f7d8819e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8d32e700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d88102c60 0x7f7d8819e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8d32e700 1 -- 172.21.15.106:0/1814391918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d881022c0 msgr2=0x7f7d88197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8d32e700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d881022c0 0x7f7d88197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8d32e700 1 -- 172.21.15.106:0/1814391918 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8819ef70 con 0x7f7d881010c0 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.081+0000 7f7d8d32e700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 0x7f7d88197310 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800f710 tx=0x7f7d78009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.082+0000 7f7d7e7fc700 1 -- 172.21.15.106:0/1814391918 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78009e60 con 0x7f7d881010c0 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.082+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8819f200 con 0x7f7d881010c0 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.082+0000 7f7d7e7fc700 1 -- 172.21.15.106:0/1814391918 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d78003960 con 0x7f7d881010c0 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.082+0000 7f7d7e7fc700 1 -- 172.21.15.106:0/1814391918 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78005560 con 0x7f7d881010c0 2024-03-28T02:10:42.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.082+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8819f6e0 con 0x7f7d881010c0 2024-03-28T02:10:42.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.083+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d88068e50 con 0x7f7d881010c0 2024-03-28T02:10:42.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.084+0000 7f7d7e7fc700 1 -- 172.21.15.106:0/1814391918 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7d78003b10 con 0x7f7d881010c0 2024-03-28T02:10:42.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.085+0000 7f7d7e7fc700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7d740a2890 0x7f7d740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:42.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.085+0000 7f7d8cb2d700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7d740a2890 0x7f7d740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:42.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.085+0000 7f7d7e7fc700 1 -- 172.21.15.106:0/1814391918 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f7d780cfb80 con 0x7f7d881010c0 2024-03-28T02:10:42.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.085+0000 7f7d8cb2d700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7d740a2890 0x7f7d740a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009c60 tx=0x7f7d84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:42.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.088+0000 7f7d7e7fc700 1 -- 172.21.15.106:0/1814391918 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7d78013070 con 0x7f7d881010c0 2024-03-28T02:10:42.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.292+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d880625d0 con 0x7f7d740a2890 2024-03-28T02:10:42.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.305+0000 7f7d7e7fc700 1 -- 172.21.15.106:0/1814391918 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f7d880625d0 con 0x7f7d740a2890 2024-03-28T02:10:42.299 INFO:teuthology.orchestra.run.smithi106.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T02:10:42.299 INFO:teuthology.orchestra.run.smithi106.stdout:mds.a smithi106 running (2m) 19s ago 2m 18.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 26f52b2b4750 2024-03-28T02:10:42.299 INFO:teuthology.orchestra.run.smithi106.stdout:mds.b smithi153 running (119s) 53s ago 119s 26.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 693b245038b9 2024-03-28T02:10:42.299 INFO:teuthology.orchestra.run.smithi106.stdout:mds.c smithi178 running (2m) 52s ago 2m 16.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a4878635d498 2024-03-28T02:10:42.300 INFO:teuthology.orchestra.run.smithi106.stdout:mds.d smithi106 running (2m) 19s ago 2m 16.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 b97c63748bd2 2024-03-28T02:10:42.300 INFO:teuthology.orchestra.run.smithi106.stdout:mds.e smithi153 running (2m) 53s ago 2m 17.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a4218b5047f2 2024-03-28T02:10:42.300 INFO:teuthology.orchestra.run.smithi106.stdout:mds.f smithi178 running (2m) 52s ago 2m 19.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 e226f5b558f6 2024-03-28T02:10:42.300 INFO:teuthology.orchestra.run.smithi106.stdout:mds.g smithi106 running (2m) 19s ago 2m 15.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 bce84fdd6b88 2024-03-28T02:10:42.300 INFO:teuthology.orchestra.run.smithi106.stdout:mds.h smithi153 running (2m) 53s ago 2m 17.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 3ebf9fb3013b 2024-03-28T02:10:42.300 INFO:teuthology.orchestra.run.smithi106.stdout:mds.i smithi178 running (2m) 52s ago 2m 20.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 5eb0e9cd985a 2024-03-28T02:10:42.301 INFO:teuthology.orchestra.run.smithi106.stdout:mds.j smithi106 running (2m) 19s ago 2m 16.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a89b9033c742 2024-03-28T02:10:42.301 INFO:teuthology.orchestra.run.smithi106.stdout:mds.k smithi153 running (2m) 53s ago 2m 15.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 87cffb547bf8 2024-03-28T02:10:42.301 INFO:teuthology.orchestra.run.smithi106.stdout:mds.l smithi178 running (2m) 52s ago 2m 17.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d8f3d1052066 2024-03-28T02:10:42.301 INFO:teuthology.orchestra.run.smithi106.stdout:mgr.x smithi106 *:9283,8765 running (9m) 19s ago 9m 514M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 392dee3cdfe6 2024-03-28T02:10:42.301 INFO:teuthology.orchestra.run.smithi106.stdout:mgr.y smithi153 *:8443,8765 running (7m) 53s ago 7m 452M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 9442b8a6cda5 2024-03-28T02:10:42.302 INFO:teuthology.orchestra.run.smithi106.stdout:mgr.z smithi178 *:8443,8765 running (7m) 52s ago 7m 452M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 334ab3dddb83 2024-03-28T02:10:42.302 INFO:teuthology.orchestra.run.smithi106.stdout:mon.a smithi106 running (9m) 19s ago 9m 50.9M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 f36f9e681e52 2024-03-28T02:10:42.302 INFO:teuthology.orchestra.run.smithi106.stdout:mon.b smithi153 running (8m) 53s ago 8m 40.4M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 03b91e3f4431 2024-03-28T02:10:42.302 INFO:teuthology.orchestra.run.smithi106.stdout:mon.c smithi178 running (8m) 52s ago 8m 40.3M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 91173284394d 2024-03-28T02:10:42.302 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 smithi106 running (7m) 19s ago 7m 48.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 5b85f5a59426 2024-03-28T02:10:42.302 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1 smithi153 running (7m) 53s ago 7m 56.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 bbb28c886025 2024-03-28T02:10:42.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2 smithi178 running (6m) 52s ago 6m 58.7M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 7f2f6d500e61 2024-03-28T02:10:42.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3 smithi106 running (6m) 19s ago 6m 55.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 7eada527844e 2024-03-28T02:10:42.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 smithi153 running (5m) 53s ago 5m 54.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 bd17c14eea9a 2024-03-28T02:10:42.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5 smithi178 running (5m) 52s ago 5m 57.4M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 785f9c0b6b34 2024-03-28T02:10:42.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6 smithi106 running (5m) 19s ago 5m 44.4M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 47e2d6065bd1 2024-03-28T02:10:42.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7 smithi153 running (4m) 53s ago 4m 43.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c5da74a232df 2024-03-28T02:10:42.304 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8 smithi178 running (4m) 52s ago 4m 46.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 0c3f7c27b304 2024-03-28T02:10:42.304 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9 smithi106 running (3m) 19s ago 3m 41.7M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 663d3ed319bc 2024-03-28T02:10:42.304 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10 smithi153 running (3m) 53s ago 3m 57.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 ca71b92a28d7 2024-03-28T02:10:42.304 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11 smithi178 running (2m) 52s ago 2m 53.1M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 e794eaf1f281 2024-03-28T02:10:42.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7d740a2890 msgr2=0x7f7d740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:42.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7d740a2890 0x7f7d740a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009c60 tx=0x7f7d84009260 comp rx=0 tx=0).stop 2024-03-28T02:10:42.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 msgr2=0x7f7d88197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:42.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 0x7f7d88197310 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800f710 tx=0x7f7d78009580 comp rx=0 tx=0).stop 2024-03-28T02:10:42.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 shutdown_connections 2024-03-28T02:10:42.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d88102c60 0x7f7d8819e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7d740a2890 0x7f7d740a4d50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d881010c0 0x7f7d88197310 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 --2- 172.21.15.106:0/1814391918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d881022c0 0x7f7d88197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:42.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 >> 172.21.15.106:0/1814391918 conn(0x7f7d880fc680 msgr2=0x7f7d88105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:42.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 shutdown_connections 2024-03-28T02:10:42.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:42.308+0000 7f7d8f592700 1 -- 172.21.15.106:0/1814391918 wait complete. 2024-03-28T02:10:42.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T02:10:42.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-03-28T02:10:42.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:42 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-28T02:10:42.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T02:10:42.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-03-28T02:10:42.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:42 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-28T02:10:42.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T02:10:42.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-03-28T02:10:42.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:42 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-28T02:10:43.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T02:10:43.388 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:43.773 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:43 smithi106 ceph-mon[31965]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:43.774 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:43 smithi106 ceph-mon[31965]: pgmap v351: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:43.774 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T02:10:43.774 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-03-28T02:10:43.774 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:43 smithi106 ceph-mon[31965]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-28T02:10:43.774 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:43 smithi106 ceph-mon[31965]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T02:10:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:43 smithi178 ceph-mon[36521]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:43 smithi178 ceph-mon[36521]: pgmap v351: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T02:10:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-03-28T02:10:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:43 smithi178 ceph-mon[36521]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-28T02:10:43.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:43 smithi178 ceph-mon[36521]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T02:10:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:43 smithi153 ceph-mon[36228]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:43 smithi153 ceph-mon[36228]: pgmap v351: 129 pgs: 129 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:10:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T02:10:43.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-03-28T02:10:43.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:43 smithi153 ceph-mon[36228]: from='mgr.14152 172.21.15.106:0/2834216226' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-28T02:10:43.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:43 smithi153 ceph-mon[36228]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T02:10:44.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:44 smithi106 ceph-mon[31965]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T02:10:44.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:44 smithi106 ceph-mon[31965]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T02:10:44.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:44 smithi178 ceph-mon[36521]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T02:10:44.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:44 smithi178 ceph-mon[36521]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T02:10:44.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:44 smithi153 ceph-mon[36228]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T02:10:44.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:44 smithi153 ceph-mon[36228]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 649 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T02:10:45.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.431+0000 7fb216cf7700 1 -- 172.21.15.106:0/1742301324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2101022c0 msgr2=0x7fb210102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:45.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.431+0000 7fb216cf7700 1 --2- 172.21.15.106:0/1742301324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2101022c0 0x7fb210102720 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0099c0 tx=0x7fb1fc009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.431+0000 7fb216cf7700 1 -- 172.21.15.106:0/1742301324 shutdown_connections 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.431+0000 7fb216cf7700 1 --2- 172.21.15.106:0/1742301324 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb210102c60 0x7fb210109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.431+0000 7fb216cf7700 1 --2- 172.21.15.106:0/1742301324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2101010c0 0x7fb2101014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.431+0000 7fb216cf7700 1 --2- 172.21.15.106:0/1742301324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2101022c0 0x7fb210102720 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.431+0000 7fb216cf7700 1 -- 172.21.15.106:0/1742301324 >> 172.21.15.106:0/1742301324 conn(0x7fb2100fc660 msgr2=0x7fb2100feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.431+0000 7fb216cf7700 1 -- 172.21.15.106:0/1742301324 shutdown_connections 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.431+0000 7fb216cf7700 1 -- 172.21.15.106:0/1742301324 wait complete. 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb216cf7700 1 Processor -- start 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb216cf7700 1 -- start start 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb216cf7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2101010c0 0x7fb21006a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb216cf7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2101022c0 0x7fb21019d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb216cf7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb210102c60 0x7fb21019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb216cf7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21010cc80 con 0x7fb2101022c0 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb216cf7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21010cb00 con 0x7fb2101010c0 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb216cf7700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21010ce00 con 0x7fb210102c60 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb214a93700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2101010c0 0x7fb21006a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb20ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2101022c0 0x7fb21019d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb214a93700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2101010c0 0x7fb21006a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:44804/0 (socket says 172.21.15.106:44804) 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb214a93700 1 -- 172.21.15.106:0/4279109683 learned_addr learned my addr 172.21.15.106:0/4279109683 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.432+0000 7fb215294700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb210102c60 0x7fb21019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb214a93700 1 -- 172.21.15.106:0/4279109683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb210102c60 msgr2=0x7fb21019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb214a93700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb210102c60 0x7fb21019f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb214a93700 1 -- 172.21.15.106:0/4279109683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2101022c0 msgr2=0x7fb21019d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb214a93700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2101022c0 0x7fb21019d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb214a93700 1 -- 172.21.15.106:0/4279109683 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2101a0090 con 0x7fb2101010c0 2024-03-28T02:10:45.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb214a93700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2101010c0 0x7fb21006a9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb20400e970 tx=0x7fb20400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:45.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb20dffb700 1 -- 172.21.15.106:0/4279109683 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204004220 con 0x7fb2101010c0 2024-03-28T02:10:45.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2101a0380 con 0x7fb2101010c0 2024-03-28T02:10:45.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb20dffb700 1 -- 172.21.15.106:0/4279109683 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2040043c0 con 0x7fb2101010c0 2024-03-28T02:10:45.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb20dffb700 1 -- 172.21.15.106:0/4279109683 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204003dc0 con 0x7fb2101010c0 2024-03-28T02:10:45.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.433+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2101a08c0 con 0x7fb2101010c0 2024-03-28T02:10:45.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.434+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb210068e50 con 0x7fb2101010c0 2024-03-28T02:10:45.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.436+0000 7fb20dffb700 1 -- 172.21.15.106:0/4279109683 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb20400c7e0 con 0x7fb2101010c0 2024-03-28T02:10:45.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.437+0000 7fb20dffb700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2000a2890 0x7fb2000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:45.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.437+0000 7fb20ffff700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2000a2890 0x7fb2000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:45.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.437+0000 7fb20dffb700 1 -- 172.21.15.106:0/4279109683 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb2040cefe0 con 0x7fb2101010c0 2024-03-28T02:10:45.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.437+0000 7fb20ffff700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2000a2890 0x7fb2000a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb21006af20 tx=0x7fb1fc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:45.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.444+0000 7fb20dffb700 1 -- 172.21.15.106:0/4279109683 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb20409bce0 con 0x7fb2101010c0 2024-03-28T02:10:45.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.664+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2100625d0 con 0x7fb2000a2890 2024-03-28T02:10:45.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.670+0000 7fb20dffb700 1 -- 172.21.15.106:0/4279109683 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fb2100625d0 con 0x7fb2000a2890 2024-03-28T02:10:45.663 INFO:teuthology.orchestra.run.smithi106.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T02:10:45.663 INFO:teuthology.orchestra.run.smithi106.stdout:mds.all 12/12 56s ago 2m smithi106=a;smithi106=d;smithi106=g;smithi106=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-03-28T02:10:45.664 INFO:teuthology.orchestra.run.smithi106.stdout:mgr 3/3 56s ago 7m smithi106=x;smithi153=y;smithi178=z;count:3 2024-03-28T02:10:45.664 INFO:teuthology.orchestra.run.smithi106.stdout:mon 3/3 56s ago 8m smithi106:172.21.15.106=a;smithi153:172.21.15.153=b;smithi178:172.21.15.178=c;count:3 2024-03-28T02:10:45.664 INFO:teuthology.orchestra.run.smithi106.stdout:osd 12 56s ago - 2024-03-28T02:10:45.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2000a2890 msgr2=0x7fb2000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2000a2890 0x7fb2000a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb21006af20 tx=0x7fb1fc005fb0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2101010c0 msgr2=0x7fb21006a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2101010c0 0x7fb21006a9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb20400e970 tx=0x7fb20400ec80 comp rx=0 tx=0).stop 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 shutdown_connections 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb210102c60 0x7fb21019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2101010c0 0x7fb21006a9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb2000a2890 0x7fb2000a4d50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 --2- 172.21.15.106:0/4279109683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2101022c0 0x7fb21019d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.672+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 >> 172.21.15.106:0/4279109683 conn(0x7fb2100fc660 msgr2=0x7fb2101055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.673+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 shutdown_connections 2024-03-28T02:10:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:45.673+0000 7fb216cf7700 1 -- 172.21.15.106:0/4279109683 wait complete. 2024-03-28T02:10:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:45 smithi178 ceph-mon[36521]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-28T02:10:45.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:45 smithi153 ceph-mon[36228]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-28T02:10:46.055 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:45 smithi106 ceph-mon[31965]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-28T02:10:46.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T02:10:46.736 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:46.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:46 smithi106 ceph-mon[31965]: from='client.25172 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:46.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:46 smithi106 ceph-mon[31965]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:10:46.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:46 smithi178 ceph-mon[36521]: from='client.25172 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:46.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:46 smithi178 ceph-mon[36521]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:10:46.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:46 smithi153 ceph-mon[36228]: from='client.25172 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:46.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:46 smithi153 ceph-mon[36228]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:10:48.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.482+0000 7f49dd958700 1 -- 172.21.15.106:0/3953971627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d8102c60 msgr2=0x7f49d8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:48.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.482+0000 7f49dd958700 1 --2- 172.21.15.106:0/3953971627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d8102c60 0x7f49d8109510 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f49d000b0d0 tx=0x7f49d000b3e0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.482+0000 7f49dd958700 1 -- 172.21.15.106:0/3953971627 shutdown_connections 2024-03-28T02:10:48.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.482+0000 7f49dd958700 1 --2- 172.21.15.106:0/3953971627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d8102c60 0x7f49d8109510 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.482+0000 7f49dd958700 1 --2- 172.21.15.106:0/3953971627 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49d81010c0 0x7f49d81014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.482+0000 7f49dd958700 1 --2- 172.21.15.106:0/3953971627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49d81022c0 0x7f49d8102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.482+0000 7f49dd958700 1 -- 172.21.15.106:0/3953971627 >> 172.21.15.106:0/3953971627 conn(0x7f49d80fc660 msgr2=0x7f49d80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.482+0000 7f49dd958700 1 -- 172.21.15.106:0/3953971627 shutdown_connections 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.482+0000 7f49dd958700 1 -- 172.21.15.106:0/3953971627 wait complete. 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49dd958700 1 Processor -- start 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49dd958700 1 -- start start 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49dd958700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d81010c0 0x7f49d81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49dd958700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49d81022c0 0x7f49d8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49dd958700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49d8102c60 0x7f49d819f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49dd958700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d810ccb0 con 0x7f49d81010c0 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49dd958700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d810cb30 con 0x7f49d8102c60 2024-03-28T02:10:48.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49dd958700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d810ce30 con 0x7f49d81022c0 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49d6ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d81010c0 0x7f49d81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49d6ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d81010c0 0x7f49d81983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50000/0 (socket says 172.21.15.106:50000) 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49d6ffd700 1 -- 172.21.15.106:0/1890037917 learned_addr learned my addr 172.21.15.106:0/1890037917 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49d77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49d8102c60 0x7f49d819f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.483+0000 7f49d67fc700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49d81022c0 0x7f49d8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49d77fe700 1 -- 172.21.15.106:0/1890037917 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49d81022c0 msgr2=0x7f49d8198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49d77fe700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49d81022c0 0x7f49d8198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49d77fe700 1 -- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d81010c0 msgr2=0x7f49d81983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49d77fe700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d81010c0 0x7f49d81983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49d77fe700 1 -- 172.21.15.106:0/1890037917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d81a0020 con 0x7f49d8102c60 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49d6ffd700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d81010c0 0x7f49d81983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49d77fe700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49d8102c60 0x7f49d819f980 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f49d000b0d0 tx=0x7f49d0007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:48.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49dc956700 1 -- 172.21.15.106:0/1890037917 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0021020 con 0x7f49d8102c60 2024-03-28T02:10:48.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49dc956700 1 -- 172.21.15.106:0/1890037917 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49d0004c50 con 0x7f49d8102c60 2024-03-28T02:10:48.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49dc956700 1 -- 172.21.15.106:0/1890037917 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d00052b0 con 0x7f49d8102c60 2024-03-28T02:10:48.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d81a02b0 con 0x7f49d8102c60 2024-03-28T02:10:48.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.484+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d81a0710 con 0x7f49d8102c60 2024-03-28T02:10:48.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.485+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d8068e50 con 0x7f49d8102c60 2024-03-28T02:10:48.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.486+0000 7f49dc956700 1 -- 172.21.15.106:0/1890037917 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f49d0024020 con 0x7f49d8102c60 2024-03-28T02:10:48.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.486+0000 7f49dc956700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f49c40a2910 0x7f49c40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:48.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.487+0000 7f49d6ffd700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f49c40a2910 0x7f49c40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:48.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.487+0000 7f49dc956700 1 -- 172.21.15.106:0/1890037917 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f49d00d0200 con 0x7f49d8102c60 2024-03-28T02:10:48.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.487+0000 7f49d6ffd700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f49c40a2910 0x7f49c40a4dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f49c0005950 tx=0x7f49c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:48.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.490+0000 7f49dc956700 1 -- 172.21.15.106:0/1890037917 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f49d002b020 con 0x7f49d8102c60 2024-03-28T02:10:48.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.694+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f49d80fe800 con 0x7f49c40a2910 2024-03-28T02:10:48.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.696+0000 7f49dc956700 1 -- 172.21.15.106:0/1890037917 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f49d80fe800 con 0x7f49c40a2910 2024-03-28T02:10:48.689 INFO:teuthology.orchestra.run.smithi106.stdout:HOST ADDR LABELS STATUS 2024-03-28T02:10:48.689 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106 172.21.15.106 2024-03-28T02:10:48.689 INFO:teuthology.orchestra.run.smithi106.stdout:smithi153 172.21.15.153 2024-03-28T02:10:48.689 INFO:teuthology.orchestra.run.smithi106.stdout:smithi178 172.21.15.178 2024-03-28T02:10:48.689 INFO:teuthology.orchestra.run.smithi106.stdout:3 hosts in cluster 2024-03-28T02:10:48.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f49c40a2910 msgr2=0x7f49c40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:48.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f49c40a2910 0x7f49c40a4dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f49c0005950 tx=0x7f49c0009260 comp rx=0 tx=0).stop 2024-03-28T02:10:48.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49d8102c60 msgr2=0x7f49d819f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:48.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49d8102c60 0x7f49d819f980 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f49d000b0d0 tx=0x7f49d0007a20 comp rx=0 tx=0).stop 2024-03-28T02:10:48.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 shutdown_connections 2024-03-28T02:10:48.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49d8102c60 0x7f49d819f980 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f49c40a2910 0x7f49c40a4dd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d81010c0 0x7f49d81983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 --2- 172.21.15.106:0/1890037917 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49d81022c0 0x7f49d8198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:48.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 >> 172.21.15.106:0/1890037917 conn(0x7f49d80fc660 msgr2=0x7f49d80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:48.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 shutdown_connections 2024-03-28T02:10:48.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:48.698+0000 7f49dd958700 1 -- 172.21.15.106:0/1890037917 wait complete. 2024-03-28T02:10:49.411 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T02:10:49.654 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:49.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:49 smithi153 ceph-mon[36228]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-03-28T02:10:49.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:49 smithi106 ceph-mon[31965]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-03-28T02:10:49.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:49 smithi178 ceph-mon[36521]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-03-28T02:10:50.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:50 smithi153 ceph-mon[36228]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:50.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:50 smithi106 ceph-mon[31965]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:50.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:50 smithi178 ceph-mon[36521]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:51.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.422+0000 7f0a06865700 1 -- 172.21.15.106:0/3044895650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a001022b0 msgr2=0x7f0a00102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:51.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.422+0000 7f0a06865700 1 --2- 172.21.15.106:0/3044895650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a001022b0 0x7f0a00102710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f09f00099c0 tx=0x7f09f0009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.422+0000 7f0a06865700 1 -- 172.21.15.106:0/3044895650 shutdown_connections 2024-03-28T02:10:51.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.422+0000 7f0a06865700 1 --2- 172.21.15.106:0/3044895650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a00102c50 0x7f0a00109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.422+0000 7f0a06865700 1 --2- 172.21.15.106:0/3044895650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001010b0 0x7f0a001014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.422+0000 7f0a06865700 1 --2- 172.21.15.106:0/3044895650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a001022b0 0x7f0a00102710 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.422+0000 7f0a06865700 1 -- 172.21.15.106:0/3044895650 >> 172.21.15.106:0/3044895650 conn(0x7f0a000fc650 msgr2=0x7f0a000fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:51.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.422+0000 7f0a06865700 1 -- 172.21.15.106:0/3044895650 shutdown_connections 2024-03-28T02:10:51.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.422+0000 7f0a06865700 1 -- 172.21.15.106:0/3044895650 wait complete. 2024-03-28T02:10:51.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.423+0000 7f0a06865700 1 Processor -- start 2024-03-28T02:10:51.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.423+0000 7f0a06865700 1 -- start start 2024-03-28T02:10:51.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.423+0000 7f0a06865700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001010b0 0x7f0a0006b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:51.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.423+0000 7f0a06865700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a001022b0 0x7f0a0019d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:51.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.423+0000 7f0a06865700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00102c50 0x7f0a0019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:51.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.423+0000 7f0a06865700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0010cc70 con 0x7f0a001022b0 2024-03-28T02:10:51.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.423+0000 7f0a06865700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0010caf0 con 0x7f0a00102c50 2024-03-28T02:10:51.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.423+0000 7f0a06865700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0010cdf0 con 0x7f0a001010b0 2024-03-28T02:10:51.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.423+0000 7f09ff7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a001022b0 0x7f0a0019d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fffff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001010b0 0x7f0a0006b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fffff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001010b0 0x7f0a0006b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:35664/0 (socket says 172.21.15.106:35664) 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fffff700 1 -- 172.21.15.106:0/1243006431 learned_addr learned my addr 172.21.15.106:0/1243006431 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09ff7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a001022b0 0x7f0a0019d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50016/0 (socket says 172.21.15.106:50016) 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f0a04e02700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00102c50 0x7f0a0019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fffff700 1 -- 172.21.15.106:0/1243006431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00102c50 msgr2=0x7f0a0019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fffff700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00102c50 0x7f0a0019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fffff700 1 -- 172.21.15.106:0/1243006431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a001022b0 msgr2=0x7f0a0019d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fffff700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a001022b0 0x7f0a0019d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fffff700 1 -- 172.21.15.106:0/1243006431 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a001a0000 con 0x7f0a001010b0 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fffff700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001010b0 0x7f0a0006b230 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09f400e970 tx=0x7f09f400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fd7fa700 1 -- 172.21.15.106:0/1243006431 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4004220 con 0x7f0a001010b0 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fd7fa700 1 -- 172.21.15.106:0/1243006431 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09f40043c0 con 0x7f0a001010b0 2024-03-28T02:10:51.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a001a02f0 con 0x7f0a001010b0 2024-03-28T02:10:51.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f09fd7fa700 1 -- 172.21.15.106:0/1243006431 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4003dc0 con 0x7f0a001010b0 2024-03-28T02:10:51.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.424+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a001a0800 con 0x7f0a001010b0 2024-03-28T02:10:51.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.425+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a00068e50 con 0x7f0a001010b0 2024-03-28T02:10:51.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.426+0000 7f09fd7fa700 1 -- 172.21.15.106:0/1243006431 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f09f400c7c0 con 0x7f0a001010b0 2024-03-28T02:10:51.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.427+0000 7f09fd7fa700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f09e80a2890 0x7f09e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:51.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.427+0000 7f09ff7fe700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f09e80a2890 0x7f09e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:51.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.427+0000 7f09fd7fa700 1 -- 172.21.15.106:0/1243006431 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f09f40cf070 con 0x7f0a001010b0 2024-03-28T02:10:51.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.427+0000 7f09ff7fe700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f09e80a2890 0x7f09e80a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f09f0000c00 tx=0x7f09f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:51.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.430+0000 7f09fd7fa700 1 -- 172.21.15.106:0/1243006431 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09f409bd10 con 0x7f0a001010b0 2024-03-28T02:10:51.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.639+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a000625d0 con 0x7f09e80a2890 2024-03-28T02:10:51.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.641+0000 7f09fd7fa700 1 -- 172.21.15.106:0/1243006431 <== mgr.14152 v2:172.21.15.106:6800/3501119323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f0a000625d0 con 0x7f09e80a2890 2024-03-28T02:10:51.634 INFO:teuthology.orchestra.run.smithi106.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T02:10:51.634 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013R400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T02:10:51.634 INFO:teuthology.orchestra.run.smithi106.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T02:10:51.634 INFO:teuthology.orchestra.run.smithi106.stdout:smithi178 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75130029480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-28T02:10:51.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f09e80a2890 msgr2=0x7f09e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:51.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f09e80a2890 0x7f09e80a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f09f0000c00 tx=0x7f09f0005fb0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001010b0 msgr2=0x7f0a0006b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:51.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001010b0 0x7f0a0006b230 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09f400e970 tx=0x7f09f400ec80 comp rx=0 tx=0).stop 2024-03-28T02:10:51.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 shutdown_connections 2024-03-28T02:10:51.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00102c50 0x7f0a0019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001010b0 0x7f0a0006b230 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f09e80a2890 0x7f09e80a4d50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 --2- 172.21.15.106:0/1243006431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a001022b0 0x7f0a0019d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:51.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 >> 172.21.15.106:0/1243006431 conn(0x7f0a000fc650 msgr2=0x7f0a00105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:51.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 shutdown_connections 2024-03-28T02:10:51.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:51.643+0000 7f0a06865700 1 -- 172.21.15.106:0/1243006431 wait complete. 2024-03-28T02:10:51.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:51 smithi153 ceph-mon[36228]: pgmap v357: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T02:10:51.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:51 smithi153 ceph-mon[36228]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-28T02:10:51.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:51 smithi153 ceph-mon[36228]: Cluster is now healthy 2024-03-28T02:10:51.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:51 smithi106 ceph-mon[31965]: pgmap v357: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T02:10:51.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:51 smithi106 ceph-mon[31965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-28T02:10:51.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:51 smithi106 ceph-mon[31965]: Cluster is now healthy 2024-03-28T02:10:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:51 smithi178 ceph-mon[36521]: pgmap v357: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T02:10:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:51 smithi178 ceph-mon[36521]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-28T02:10:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:51 smithi178 ceph-mon[36521]: Cluster is now healthy 2024-03-28T02:10:52.361 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T02:10:52.369 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:10:52.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T02:10:52.613 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:53.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:53 smithi153 ceph-mon[36228]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:53.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:53 smithi153 ceph-mon[36228]: pgmap v358: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-28T02:10:53.810 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:53 smithi106 ceph-mon[31965]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:53.810 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:53 smithi106 ceph-mon[31965]: pgmap v358: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-28T02:10:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:53 smithi178 ceph-mon[36521]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:10:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:53 smithi178 ceph-mon[36521]: pgmap v358: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 -- 172.21.15.106:0/3457811545 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31601021c0 msgr2=0x7f31601025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 --2- 172.21.15.106:0/3457811545 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31601021c0 0x7f31601025e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f31500098f0 tx=0x7f3150009c00 comp rx=0 tx=0).stop 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 -- 172.21.15.106:0/3457811545 shutdown_connections 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 --2- 172.21.15.106:0/3457811545 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3160103d60 0x7f316010a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 --2- 172.21.15.106:0/3457811545 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31601021c0 0x7f31601025e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 --2- 172.21.15.106:0/3457811545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31601033c0 0x7f3160103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 -- 172.21.15.106:0/3457811545 >> 172.21.15.106:0/3457811545 conn(0x7f31600fd7a0 msgr2=0x7f31600ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 -- 172.21.15.106:0/3457811545 shutdown_connections 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 -- 172.21.15.106:0/3457811545 wait complete. 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.363+0000 7f3164ad2700 1 Processor -- start 2024-03-28T02:10:54.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f3164ad2700 1 -- start start 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f3164ad2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31601021c0 0x7f3160075ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f3164ad2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31601033c0 0x7f3160074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f3164ad2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3160103d60 0x7f3160074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f3164ad2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316010ddc0 con 0x7f3160103d60 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f3164ad2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316010dc40 con 0x7f31601033c0 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f3164ad2700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316010df40 con 0x7f31601021c0 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315ed9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3160103d60 0x7f3160074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315dd9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31601033c0 0x7f3160074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315ed9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3160103d60 0x7f3160074ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50032/0 (socket says 172.21.15.106:50032) 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315dd9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31601033c0 0x7f3160074580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:35076/0 (socket says 172.21.15.106:35076) 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315ed9d700 1 -- 172.21.15.106:0/1083594603 learned_addr learned my addr 172.21.15.106:0/1083594603 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315e59c700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31601021c0 0x7f3160075ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315e59c700 1 -- 172.21.15.106:0/1083594603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31601033c0 msgr2=0x7f3160074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315e59c700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31601033c0 0x7f3160074580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315e59c700 1 -- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3160103d60 msgr2=0x7f3160074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:54.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315e59c700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3160103d60 0x7f3160074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315e59c700 1 -- 172.21.15.106:0/1083594603 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3160075390 con 0x7f31601021c0 2024-03-28T02:10:54.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315ed9d700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3160103d60 0x7f3160074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:54.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f315e59c700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31601021c0 0x7f3160075ee0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3150000c00 tx=0x7f31500052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:54.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.364+0000 7f314f7fe700 1 -- 172.21.15.106:0/1083594603 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315000b560 con 0x7f31601021c0 2024-03-28T02:10:54.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.365+0000 7f314f7fe700 1 -- 172.21.15.106:0/1083594603 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f315000bbe0 con 0x7f31601021c0 2024-03-28T02:10:54.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.365+0000 7f314f7fe700 1 -- 172.21.15.106:0/1083594603 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150020410 con 0x7f31601021c0 2024-03-28T02:10:54.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.365+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31601a25c0 con 0x7f31601021c0 2024-03-28T02:10:54.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.365+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31601a2ac0 con 0x7f31601021c0 2024-03-28T02:10:54.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.365+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3160068e50 con 0x7f31601021c0 2024-03-28T02:10:54.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.367+0000 7f314f7fe700 1 -- 172.21.15.106:0/1083594603 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3150003680 con 0x7f31601021c0 2024-03-28T02:10:54.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.367+0000 7f314f7fe700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31480a2910 0x7f31480a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:54.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.367+0000 7f314f7fe700 1 -- 172.21.15.106:0/1083594603 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f31500cf950 con 0x7f31601021c0 2024-03-28T02:10:54.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.367+0000 7f315dd9b700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31480a2910 0x7f31480a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:54.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.368+0000 7f315dd9b700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31480a2910 0x7f31480a4dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3154009c60 tx=0x7f3154009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:54.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.371+0000 7f314f7fe700 1 -- 172.21.15.106:0/1083594603 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3150013030 con 0x7f31601021c0 2024-03-28T02:10:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.611+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f316002de50 con 0x7f31601021c0 2024-03-28T02:10:54.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.612+0000 7f314f7fe700 1 -- 172.21.15.106:0/1083594603 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+3048 (secure 0 0 0) 0x7f315009c680 con 0x7f31601021c0 2024-03-28T02:10:54.605 INFO:teuthology.orchestra.run.smithi106.stdout:e28 2024-03-28T02:10:54.605 INFO:teuthology.orchestra.run.smithi106.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:legacy client fscid: 1 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:Filesystem 'cephfs' (1) 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:fs_name cephfs 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:epoch 28 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:created 2024-03-28T02:08:48.782692+0000 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:modified 2024-03-28T02:09:11.288485+0000 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:tableserver 0 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:root 0 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:session_timeout 300 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:session_autoclose 300 2024-03-28T02:10:54.606 INFO:teuthology.orchestra.run.smithi106.stdout:max_file_size 1099511627776 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:max_xattr_size 65536 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:required_client_features {} 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:last_failure 0 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:last_failure_osd_epoch 0 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:max_mds 5 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:in 0,1,2,3,4 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:up {0=24509,1=24499,2=24460,3=24446,4=24415} 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:failed 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:damaged 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:stopped 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:data_pools [3,4] 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:metadata_pool 2 2024-03-28T02:10:54.607 INFO:teuthology.orchestra.run.smithi106.stdout:inline_data disabled 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:balancer 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:bal_rank_mask -1 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:standby_count_wanted 0 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:qdb_cluster leader: 24509 members: 24415,24499,24509,24446,24460 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.b{0:24509} state up:active seq 3 addr [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.h{0:24479} state up:standby-replay seq 1 addr [v2:172.21.15.153:6836/1253460068,v1:172.21.15.153:6837/1253460068] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.i{1:24499} state up:active seq 10 addr [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.c{1:24472} state up:standby-replay seq 1 addr [v2:172.21.15.178:6836/3554528514,v1:172.21.15.178:6837/3554528514] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.f{2:24460} state up:active seq 12 addr [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.k{2:24404} state up:standby-replay seq 1 addr [v2:172.21.15.153:6832/564442660,v1:172.21.15.153:6833/564442660] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.e{3:24446} state up:active seq 13 addr [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.a{3:14637} state up:standby-replay seq 1 addr [v2:172.21.15.106:6840/1766684450,v1:172.21.15.106:6841/1766684450] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.l{4:24415} state up:active seq 16 addr [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.g{4:14625} state up:standby-replay seq 1 addr [v2:172.21.15.106:6838/1515774363,v1:172.21.15.106:6839/1515774363] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.608 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:54.609 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:54.609 INFO:teuthology.orchestra.run.smithi106.stdout:Standby daemons: 2024-03-28T02:10:54.609 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:54.609 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.j{-1:14529} state up:standby seq 1 addr [v2:172.21.15.106:6834/3072695267,v1:172.21.15.106:6835/3072695267] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.609 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.d{-1:14547} state up:standby seq 1 addr [v2:172.21.15.106:6836/497398504,v1:172.21.15.106:6837/497398504] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T02:10:54.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31480a2910 msgr2=0x7f31480a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31480a2910 0x7f31480a4dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3154009c60 tx=0x7f3154009260 comp rx=0 tx=0).stop 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31601021c0 msgr2=0x7f3160075ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31601021c0 0x7f3160075ee0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3150000c00 tx=0x7f31500052f0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 shutdown_connections 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f31480a2910 0x7f31480a4dd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3160103d60 0x7f3160074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31601021c0 0x7f3160075ee0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 --2- 172.21.15.106:0/1083594603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31601033c0 0x7f3160074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 >> 172.21.15.106:0/1083594603 conn(0x7f31600fd7a0 msgr2=0x7f31601067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 shutdown_connections 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:54.615+0000 7f3164ad2700 1 -- 172.21.15.106:0/1083594603 wait complete. 2024-03-28T02:10:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 28 2024-03-28T02:10:55.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T02:10:55.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:55 smithi106 ceph-mon[31965]: pgmap v359: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-03-28T02:10:55.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:55 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1083594603' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T02:10:55.599 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:10:55.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:55 smithi153 ceph-mon[36228]: pgmap v359: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-03-28T02:10:55.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:55 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1083594603' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T02:10:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:55 smithi178 ceph-mon[36521]: pgmap v359: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-03-28T02:10:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:55 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1083594603' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.360+0000 7fd147282700 1 -- 172.21.15.106:0/3281441028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1401033f0 msgr2=0x7fd140103850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.360+0000 7fd147282700 1 --2- 172.21.15.106:0/3281441028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1401033f0 0x7fd140103850 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd1300099c0 tx=0x7fd130009cd0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.360+0000 7fd147282700 1 -- 172.21.15.106:0/3281441028 shutdown_connections 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.360+0000 7fd147282700 1 --2- 172.21.15.106:0/3281441028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd140103d90 0x7fd14010a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.360+0000 7fd147282700 1 --2- 172.21.15.106:0/3281441028 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1401021f0 0x7fd140102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.360+0000 7fd147282700 1 --2- 172.21.15.106:0/3281441028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1401033f0 0x7fd140103850 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.360+0000 7fd147282700 1 -- 172.21.15.106:0/3281441028 >> 172.21.15.106:0/3281441028 conn(0x7fd1400fd790 msgr2=0x7fd1400ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.360+0000 7fd147282700 1 -- 172.21.15.106:0/3281441028 shutdown_connections 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.360+0000 7fd147282700 1 -- 172.21.15.106:0/3281441028 wait complete. 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd147282700 1 Processor -- start 2024-03-28T02:10:57.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd147282700 1 -- start start 2024-03-28T02:10:57.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd147282700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1401021f0 0x7fd1401983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:57.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd147282700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1401033f0 0x7fd140198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:57.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd147282700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd140103d90 0x7fd14019f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:57.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd147282700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14010ddb0 con 0x7fd140103d90 2024-03-28T02:10:57.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd147282700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14010dc30 con 0x7fd1401021f0 2024-03-28T02:10:57.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd147282700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14010df30 con 0x7fd1401033f0 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd14581f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd140103d90 0x7fd14019f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd14481d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1401033f0 0x7fd140198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd14581f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd140103d90 0x7fd14019f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50058/0 (socket says 172.21.15.106:50058) 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd14581f700 1 -- 172.21.15.106:0/4207189094 learned_addr learned my addr 172.21.15.106:0/4207189094 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd14501e700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1401021f0 0x7fd1401983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.361+0000 7fd14581f700 1 -- 172.21.15.106:0/4207189094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1401033f0 msgr2=0x7fd140198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd14581f700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1401033f0 0x7fd140198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd14581f700 1 -- 172.21.15.106:0/4207189094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1401021f0 msgr2=0x7fd1401983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd14581f700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1401021f0 0x7fd1401983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd14581f700 1 -- 172.21.15.106:0/4207189094 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400743c0 con 0x7fd140103d90 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd14481d700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1401033f0 0x7fd140198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd14581f700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd140103d90 0x7fd14019f990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00e970 tx=0x7fd13c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd1367fc700 1 -- 172.21.15.106:0/4207189094 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c004a30 con 0x7fd140103d90 2024-03-28T02:10:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd1367fc700 1 -- 172.21.15.106:0/4207189094 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd13c005500 con 0x7fd140103d90 2024-03-28T02:10:57.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1400746b0 con 0x7fd140103d90 2024-03-28T02:10:57.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd1367fc700 1 -- 172.21.15.106:0/4207189094 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c00cdc0 con 0x7fd140103d90 2024-03-28T02:10:57.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.362+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd140074ba0 con 0x7fd140103d90 2024-03-28T02:10:57.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.363+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd140068e50 con 0x7fd140103d90 2024-03-28T02:10:57.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.364+0000 7fd1367fc700 1 -- 172.21.15.106:0/4207189094 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd13c005000 con 0x7fd140103d90 2024-03-28T02:10:57.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.364+0000 7fd1367fc700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd12c0a2890 0x7fd12c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:57.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.365+0000 7fd14501e700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd12c0a2890 0x7fd12c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:57.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.365+0000 7fd1367fc700 1 -- 172.21.15.106:0/4207189094 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd13c0cf300 con 0x7fd140103d90 2024-03-28T02:10:57.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.365+0000 7fd14501e700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd12c0a2890 0x7fd12c0a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd140103250 tx=0x7fd13800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:57.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.368+0000 7fd1367fc700 1 -- 172.21.15.106:0/4207189094 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd13c053740 con 0x7fd140103d90 2024-03-28T02:10:57.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.568+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fd140075fe0 con 0x7fd140103d90 2024-03-28T02:10:57.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:57 smithi106 ceph-mon[31965]: pgmap v360: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-03-28T02:10:57.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.569+0000 7fd1367fc700 1 -- 172.21.15.106:0/4207189094 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) v1 ==== 56+0+5688 (secure 0 0 0) 0x7fd140075fe0 con 0x7fd140103d90 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:epoch 93 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:fsid d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:created 2024-03-28T02:00:42.806452+0000 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:modified 2024-03-28T02:10:43.590748+0000 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:crush_version 27 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:full_ratio 0.95 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:backfillfull_ratio 0.9 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:nearfull_ratio 0.85 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:require_min_compat_client luminous 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:min_compat_client luminous 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:require_osd_release squid 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:stretch_mode_enabled false 2024-03-28T02:10:57.563 INFO:teuthology.orchestra.run.smithi106.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:max_osd 12 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6802/1092015924,v1:172.21.15.106:6803/1092015924] [v2:172.21.15.106:6804/1092015924,v1:172.21.15.106:6805/1092015924] exists,up 57987f85-e72f-4e07-b777-cc2243ffbbe7 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/2006532693,v1:172.21.15.153:6801/2006532693] [v2:172.21.15.153:6802/2006532693,v1:172.21.15.153:6803/2006532693] exists,up 45e0b5df-6da6-4f24-aaf4-2d1531425742 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6800/3105645780,v1:172.21.15.178:6801/3105645780] [v2:172.21.15.178:6802/3105645780,v1:172.21.15.178:6803/3105645780] exists,up 44ba4df3-ae2f-4947-8eb9-e7d91e69bba2 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6810/3263525761,v1:172.21.15.106:6811/3263525761] [v2:172.21.15.106:6812/3263525761,v1:172.21.15.106:6813/3263525761] exists,up 21a1eac9-0d56-4b09-914d-7a70444b90ed 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/3126868610,v1:172.21.15.153:6809/3126868610] [v2:172.21.15.153:6810/3126868610,v1:172.21.15.153:6811/3126868610] exists,up ad7fd91b-fc97-464b-bd5b-56e9bcf663fe 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6808/3071904021,v1:172.21.15.178:6809/3071904021] [v2:172.21.15.178:6810/3071904021,v1:172.21.15.178:6811/3071904021] exists,up 8d9710de-3c2e-492e-8a73-e8483bc7b269 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6818/233665499,v1:172.21.15.106:6819/233665499] [v2:172.21.15.106:6820/233665499,v1:172.21.15.106:6821/233665499] exists,up 063f0b59-1636-4cff-8de5-176afcb329b6 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/1996966525,v1:172.21.15.153:6817/1996966525] [v2:172.21.15.153:6818/1996966525,v1:172.21.15.153:6819/1996966525] exists,up 77f29280-5275-445e-93a1-c93e64f17496 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6816/3897935183,v1:172.21.15.178:6817/3897935183] [v2:172.21.15.178:6818/3897935183,v1:172.21.15.178:6819/3897935183] exists,up 46ffe893-a42e-4374-a83a-1316c70cd49d 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6826/4082459972,v1:172.21.15.106:6827/4082459972] [v2:172.21.15.106:6828/4082459972,v1:172.21.15.106:6829/4082459972] exists,up 16014c57-a839-460f-b9e5-e1a336f53c52 2024-03-28T02:10:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/1470813442,v1:172.21.15.153:6825/1470813442] [v2:172.21.15.153:6826/1470813442,v1:172.21.15.153:6827/1470813442] exists,up 5d65fed3-ba4f-4a3c-9469-c624012d5fa4 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6824/3563501729,v1:172.21.15.178:6825/3563501729] [v2:172.21.15.178:6826/3563501729,v1:172.21.15.178:6827/3563501729] exists,up 37920df5-7610-4acf-98e8-33fad58f4d2e 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 2.b [2,4] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 2.10 [8,11] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 2.13 [2,11] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 3.c [3,9] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 3.11 [4,6] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 3.19 [3,7] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 3.1b [4,8,5,9] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 3.3c [4,9] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 4.5 [9,10] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 4.f [9,8] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 4.16 [9,8] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:pg_upmap_items 4.1b [5,8] 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/11266040 expires 2024-03-29T02:01:42.267941+0000 2024-03-28T02:10:57.565 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/2884389364 expires 2024-03-29T02:01:42.267941+0000 2024-03-28T02:10:57.566 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/4050392782 expires 2024-03-29T02:01:42.267941+0000 2024-03-28T02:10:57.566 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:6800/1510653090 expires 2024-03-29T02:01:15.853032+0000 2024-03-28T02:10:57.566 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/103827077 expires 2024-03-29T02:01:15.853032+0000 2024-03-28T02:10:57.566 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:6801/305853075 expires 2024-03-29T02:01:42.267941+0000 2024-03-28T02:10:57.566 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/256234961 expires 2024-03-29T02:01:15.853032+0000 2024-03-28T02:10:57.566 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:6800/305853075 expires 2024-03-29T02:01:42.267941+0000 2024-03-28T02:10:57.566 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:6801/1510653090 expires 2024-03-29T02:01:15.853032+0000 2024-03-28T02:10:57.566 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/2759478196 expires 2024-03-29T02:01:15.853032+0000 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.572+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd12c0a2890 msgr2=0x7fd12c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.572+0000 7fd147282700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd12c0a2890 0x7fd12c0a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd140103250 tx=0x7fd13800b040 comp rx=0 tx=0).stop 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.572+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd140103d90 msgr2=0x7fd14019f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.572+0000 7fd147282700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd140103d90 0x7fd14019f990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00e970 tx=0x7fd13c00ec80 comp rx=0 tx=0).stop 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.573+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 shutdown_connections 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.573+0000 7fd147282700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fd12c0a2890 0x7fd12c0a4d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.573+0000 7fd147282700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd140103d90 0x7fd14019f990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.573+0000 7fd147282700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1401021f0 0x7fd1401983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.573+0000 7fd147282700 1 --2- 172.21.15.106:0/4207189094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1401033f0 0x7fd140198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.573+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 >> 172.21.15.106:0/4207189094 conn(0x7fd1400fd790 msgr2=0x7fd140106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.573+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 shutdown_connections 2024-03-28T02:10:57.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:57.573+0000 7fd147282700 1 -- 172.21.15.106:0/4207189094 wait complete. 2024-03-28T02:10:57.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:57 smithi153 ceph-mon[36228]: pgmap v360: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-03-28T02:10:57.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:57 smithi178 ceph-mon[36521]: pgmap v360: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-03-28T02:10:58.298 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T02:10:58.309 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:10:58.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.422+0000 7f565d8f8640 1 -- 172.21.15.106:0/593862709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581005c0 msgr2=0x7f56581009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.422+0000 7f565d8f8640 1 --2- 172.21.15.106:0/593862709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581005c0 0x7f56581009e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5648009900 tx=0x7f564802bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:58.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 -- 172.21.15.106:0/593862709 shutdown_connections 2024-03-28T02:10:58.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 --2- 172.21.15.106:0/593862709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5658102170 0x7f5658108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 --2- 172.21.15.106:0/593862709 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56581017b0 0x7f5658101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 --2- 172.21.15.106:0/593862709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581005c0 0x7f56581009e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 -- 172.21.15.106:0/593862709 >> 172.21.15.106:0/593862709 conn(0x7f56580fbb30 msgr2=0x7f56580fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:58.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 -- 172.21.15.106:0/593862709 shutdown_connections 2024-03-28T02:10:58.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 -- 172.21.15.106:0/593862709 wait complete. 2024-03-28T02:10:58.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 Processor -- start 2024-03-28T02:10:58.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 -- start start 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581005c0 0x7f5658197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581017b0 0x7f5658197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5658102170 0x7f565819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565810c250 con 0x7f56581017b0 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565810c0d0 con 0x7f56581005c0 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f565d8f8640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565810c3d0 con 0x7f5658102170 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.423+0000 7f564ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581017b0 0x7f5658197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f56577fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5658102170 0x7f565819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f5656ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581005c0 0x7f5658197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f564ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581017b0 0x7f5658197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54490/0 (socket says 172.21.15.106:54490) 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f564ffff640 1 -- 172.21.15.106:0/3325087637 learned_addr learned my addr 172.21.15.106:0/3325087637 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:58.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f564ffff640 1 -- 172.21.15.106:0/3325087637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5658102170 msgr2=0x7f565819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f564ffff640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5658102170 0x7f565819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f564ffff640 1 -- 172.21.15.106:0/3325087637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581005c0 msgr2=0x7f5658197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f564ffff640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581005c0 0x7f5658197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f564ffff640 1 -- 172.21.15.106:0/3325087637 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565819f630 con 0x7f56581017b0 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f564ffff640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581017b0 0x7f5658197f50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f564000e9f0 tx=0x7f564000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f565c8f6640 1 -- 172.21.15.106:0/3325087637 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564000cd80 con 0x7f56581017b0 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f565c8f6640 1 -- 172.21.15.106:0/3325087637 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5640005500 con 0x7f56581017b0 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565819f920 con 0x7f56581017b0 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f565c8f6640 1 -- 172.21.15.106:0/3325087637 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640010690 con 0x7f56581017b0 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.424+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565819fe20 con 0x7f56581017b0 2024-03-28T02:10:58.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.425+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56581a00b0 con 0x7f56581017b0 2024-03-28T02:10:58.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.426+0000 7f565c8f6640 1 -- 172.21.15.106:0/3325087637 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f56400046c0 con 0x7f56581017b0 2024-03-28T02:10:58.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.427+0000 7f565c8f6640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56440a2800 0x7f56440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:58.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.427+0000 7f5656ffd640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56440a2800 0x7f56440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:58.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.427+0000 7f565c8f6640 1 -- 172.21.15.106:0/3325087637 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f56400cf1f0 con 0x7f56581017b0 2024-03-28T02:10:58.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.427+0000 7f5656ffd640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56440a2800 0x7f56440a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5648000c00 tx=0x7f5648005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:58.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.431+0000 7f565c8f6640 1 -- 172.21.15.106:0/3325087637 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f564009bef0 con 0x7f56581017b0 2024-03-28T02:10:58.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:58 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4207189094' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T02:10:58.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.609+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56581a0440 con 0x7f56581017b0 2024-03-28T02:10:58.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.611+0000 7f565c8f6640 1 -- 172.21.15.106:0/3325087637 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24845 (secure 0 0 0) 0x7f56581a0440 con 0x7f56581017b0 2024-03-28T02:10:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:10:58.604 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":93,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:10:43.590748+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:10:58.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56440a2800 msgr2=0x7f56440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56440a2800 0x7f56440a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5648000c00 tx=0x7f5648005fd0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581017b0 msgr2=0x7f5658197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581017b0 0x7f5658197f50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f564000e9f0 tx=0x7f564000edf0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 shutdown_connections 2024-03-28T02:10:58.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5658102170 0x7f565819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.606 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56581017b0 0x7f5658197f50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.607 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f56440a2800 0x7f56440a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.607 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 --2- 172.21.15.106:0/3325087637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581005c0 0x7f5658197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.607 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 >> 172.21.15.106:0/3325087637 conn(0x7f56580fbb30 msgr2=0x7f56580fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:58.607 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 shutdown_connections 2024-03-28T02:10:58.607 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.613+0000 7f565d8f8640 1 -- 172.21.15.106:0/3325087637 wait complete. 2024-03-28T02:10:58.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T02:08:46.952662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T02:08:47.976273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T02:08:51.703592+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T02:10:58.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:10:58.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.730+0000 7fa0af9a9640 1 -- 172.21.15.106:0/558667078 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a8103270 msgr2=0x7fa0a8109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.730+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/558667078 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a8103270 0x7fa0a8109b00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009ca0 tx=0x7fa0a002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.730+0000 7fa0af9a9640 1 -- 172.21.15.106:0/558667078 shutdown_connections 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.730+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/558667078 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a8103270 0x7fa0a8109b00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.730+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/558667078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0a81028b0 0x7fa0a8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.730+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/558667078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0a81016c0 0x7fa0a8101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.730+0000 7fa0af9a9640 1 -- 172.21.15.106:0/558667078 >> 172.21.15.106:0/558667078 conn(0x7fa0a80fcc30 msgr2=0x7fa0a80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.730+0000 7fa0af9a9640 1 -- 172.21.15.106:0/558667078 shutdown_connections 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.730+0000 7fa0af9a9640 1 -- 172.21.15.106:0/558667078 wait complete. 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0af9a9640 1 Processor -- start 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0af9a9640 1 -- start start 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0af9a9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a81016c0 0x7fa0a8197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0af9a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0a81028b0 0x7fa0a8197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0af9a9640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0a8103270 0x7fa0a819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0af9a9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a810d340 con 0x7fa0a81016c0 2024-03-28T02:10:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0af9a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a810d1c0 con 0x7fa0a81028b0 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0af9a9640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a810d4c0 con 0x7fa0a8103270 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0ad745640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a81016c0 0x7fa0a8197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0ad745640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a81016c0 0x7fa0a8197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54510/0 (socket says 172.21.15.106:54510) 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0ad745640 1 -- 172.21.15.106:0/2113733214 learned_addr learned my addr 172.21.15.106:0/2113733214 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0adf46640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0a8103270 0x7fa0a819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0acf44640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0a81028b0 0x7fa0a8197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0ad745640 1 -- 172.21.15.106:0/2113733214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0a8103270 msgr2=0x7fa0a819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0ad745640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0a8103270 0x7fa0a819ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0ad745640 1 -- 172.21.15.106:0/2113733214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0a81028b0 msgr2=0x7fa0a8197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0ad745640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0a81028b0 0x7fa0a8197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0ad745640 1 -- 172.21.15.106:0/2113733214 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a819f570 con 0x7fa0a81016c0 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa0ad745640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a81016c0 0x7fa0a8197950 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400e9c0 tx=0x7fa0a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa09e7fc640 1 -- 172.21.15.106:0/2113733214 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a400cd80 con 0x7fa0a81016c0 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.731+0000 7fa09e7fc640 1 -- 172.21.15.106:0/2113733214 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0a40035f0 con 0x7fa0a81016c0 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.732+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a819f860 con 0x7fa0a81016c0 2024-03-28T02:10:58.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.732+0000 7fa09e7fc640 1 -- 172.21.15.106:0/2113733214 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4010690 con 0x7fa0a81016c0 2024-03-28T02:10:58.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.732+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a819fd60 con 0x7fa0a81016c0 2024-03-28T02:10:58.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.733+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a8108980 con 0x7fa0a81016c0 2024-03-28T02:10:58.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.734+0000 7fa09e7fc640 1 -- 172.21.15.106:0/2113733214 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa0a40037a0 con 0x7fa0a81016c0 2024-03-28T02:10:58.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.734+0000 7fa09e7fc640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa0940a2800 0x7fa0940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:58.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.734+0000 7fa0acf44640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa0940a2800 0x7fa0940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:58.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.734+0000 7fa09e7fc640 1 -- 172.21.15.106:0/2113733214 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa0a40cf230 con 0x7fa0a81016c0 2024-03-28T02:10:58.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.735+0000 7fa0acf44640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa0940a2800 0x7fa0940a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8198e60 tx=0x7fa098005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:58.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.738+0000 7fa09e7fc640 1 -- 172.21.15.106:0/2113733214 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa0a409bf30 con 0x7fa0a81016c0 2024-03-28T02:10:58.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:58 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4207189094' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T02:10:58.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:58 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4207189094' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T02:10:58.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.912+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0a802d220 con 0x7fa0a81016c0 2024-03-28T02:10:58.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.912+0000 7fa09e7fc640 1 -- 172.21.15.106:0/2113733214 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0a409b830 con 0x7fa0a81016c0 2024-03-28T02:10:58.905 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:10:58.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa0940a2800 msgr2=0x7fa0940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa0940a2800 0x7fa0940a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8198e60 tx=0x7fa098005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a81016c0 msgr2=0x7fa0a8197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a81016c0 0x7fa0a8197950 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400e9c0 tx=0x7fa0a400ee80 comp rx=0 tx=0).stop 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 shutdown_connections 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0a8103270 0x7fa0a819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa0940a2800 0x7fa0940a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0a81028b0 0x7fa0a8197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 --2- 172.21.15.106:0/2113733214 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0a81016c0 0x7fa0a8197950 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 >> 172.21.15.106:0/2113733214 conn(0x7fa0a80fcc30 msgr2=0x7fa0a8105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 shutdown_connections 2024-03-28T02:10:58.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:58.914+0000 7fa0af9a9640 1 -- 172.21.15.106:0/2113733214 wait complete. 2024-03-28T02:10:58.919 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T02:10:59.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.030+0000 7f46e48ae640 1 -- 172.21.15.106:0/3996163209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e01028a0 msgr2=0x7f46e0102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.030+0000 7f46e48ae640 1 --2- 172.21.15.106:0/3996163209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e01028a0 0x7f46e0102d20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009900 tx=0x7f46d002bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:59.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.030+0000 7f46e48ae640 1 -- 172.21.15.106:0/3996163209 shutdown_connections 2024-03-28T02:10:59.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.030+0000 7f46e48ae640 1 --2- 172.21.15.106:0/3996163209 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46e0103260 0x7f46e0109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.030+0000 7f46e48ae640 1 --2- 172.21.15.106:0/3996163209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e01028a0 0x7f46e0102d20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.030+0000 7f46e48ae640 1 --2- 172.21.15.106:0/3996163209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46e01016b0 0x7f46e0101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.030+0000 7f46e48ae640 1 -- 172.21.15.106:0/3996163209 >> 172.21.15.106:0/3996163209 conn(0x7f46e00fcc40 msgr2=0x7f46e00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:59.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.030+0000 7f46e48ae640 1 -- 172.21.15.106:0/3996163209 shutdown_connections 2024-03-28T02:10:59.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.030+0000 7f46e48ae640 1 -- 172.21.15.106:0/3996163209 wait complete. 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46e48ae640 1 Processor -- start 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46e48ae640 1 -- start start 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46e48ae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46e01016b0 0x7f46e0075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46e48ae640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46e01028a0 0x7f46e0073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46e48ae640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e0103260 0x7f46e0073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46e48ae640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e010d310 con 0x7f46e0103260 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46e48ae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e010d190 con 0x7f46e01016b0 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46e48ae640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e010d490 con 0x7f46e01028a0 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ded9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e0103260 0x7f46e0073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ded9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e0103260 0x7f46e0073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54524/0 (socket says 172.21.15.106:54524) 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ded9d640 1 -- 172.21.15.106:0/4252798309 learned_addr learned my addr 172.21.15.106:0/4252798309 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46de59c640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46e01016b0 0x7f46e0075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ddd9b640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46e01028a0 0x7f46e0073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ded9d640 1 -- 172.21.15.106:0/4252798309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46e01028a0 msgr2=0x7f46e0073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ded9d640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46e01028a0 0x7f46e0073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ded9d640 1 -- 172.21.15.106:0/4252798309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46e01016b0 msgr2=0x7f46e0075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ded9d640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46e01016b0 0x7f46e0075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ded9d640 1 -- 172.21.15.106:0/4252798309 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e0074860 con 0x7f46e0103260 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46ded9d640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e0103260 0x7f46e0073fd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009fd0 tx=0x7f46d800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.031+0000 7f46d77fe640 1 -- 172.21.15.106:0/4252798309 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8017070 con 0x7f46e0103260 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.032+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e01a1b30 con 0x7f46e0103260 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.032+0000 7f46d77fe640 1 -- 172.21.15.106:0/4252798309 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d8003e50 con 0x7f46e0103260 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.032+0000 7f46d77fe640 1 -- 172.21.15.106:0/4252798309 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d80106c0 con 0x7f46e0103260 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.032+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e01a2000 con 0x7f46e0103260 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.032+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e0108a50 con 0x7f46e0103260 2024-03-28T02:10:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.034+0000 7f46d77fe640 1 -- 172.21.15.106:0/4252798309 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f46d8003970 con 0x7f46e0103260 2024-03-28T02:10:59.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.034+0000 7f46d77fe640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46cc0a2800 0x7f46cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.034+0000 7f46de59c640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46cc0a2800 0x7f46cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.034+0000 7f46d77fe640 1 -- 172.21.15.106:0/4252798309 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f46d80cea70 con 0x7f46e0103260 2024-03-28T02:10:59.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.035+0000 7f46de59c640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46cc0a2800 0x7f46cc0a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f46e0102700 tx=0x7f46c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:59.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.039+0000 7f46d77fe640 1 -- 172.21.15.106:0/4252798309 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f46d809b120 con 0x7f46e0103260 2024-03-28T02:10:59.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.212+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f46e0062340 con 0x7f46e0103260 2024-03-28T02:10:59.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.213+0000 7f46d77fe640 1 -- 172.21.15.106:0/4252798309 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f46d809f1c0 con 0x7f46e0103260 2024-03-28T02:10:59.206 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:10:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.214+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46cc0a2800 msgr2=0x7f46cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.214+0000 7f46e48ae640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46cc0a2800 0x7f46cc0a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f46e0102700 tx=0x7f46c8005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e0103260 msgr2=0x7f46e0073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e0103260 0x7f46e0073fd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009fd0 tx=0x7f46d800b260 comp rx=0 tx=0).stop 2024-03-28T02:10:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 shutdown_connections 2024-03-28T02:10:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46e0103260 0x7f46e0073fd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46e01028a0 0x7f46e0073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46cc0a2800 0x7f46cc0a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 --2- 172.21.15.106:0/4252798309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46e01016b0 0x7f46e0075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 >> 172.21.15.106:0/4252798309 conn(0x7f46e00fcc40 msgr2=0x7f46e0105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:59.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 shutdown_connections 2024-03-28T02:10:59.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.215+0000 7f46e48ae640 1 -- 172.21.15.106:0/4252798309 wait complete. 2024-03-28T02:10:59.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T02:10:59.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.329+0000 7ff068fcd640 1 -- 172.21.15.106:0/1934771408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064102150 msgr2=0x7ff0641089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.329+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1934771408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064102150 0x7ff0641089e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009ca0 tx=0x7ff05c02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.329+0000 7ff068fcd640 1 -- 172.21.15.106:0/1934771408 shutdown_connections 2024-03-28T02:10:59.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.329+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1934771408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064102150 0x7ff0641089e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.329+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1934771408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff064101790 0x7ff064101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.329+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1934771408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0641005a0 0x7ff0641009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.329+0000 7ff068fcd640 1 -- 172.21.15.106:0/1934771408 >> 172.21.15.106:0/1934771408 conn(0x7ff0640fbb30 msgr2=0x7ff0640fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:59.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.329+0000 7ff068fcd640 1 -- 172.21.15.106:0/1934771408 shutdown_connections 2024-03-28T02:10:59.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.329+0000 7ff068fcd640 1 -- 172.21.15.106:0/1934771408 wait complete. 2024-03-28T02:10:59.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff068fcd640 1 Processor -- start 2024-03-28T02:10:59.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff068fcd640 1 -- start start 2024-03-28T02:10:59.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff068fcd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0641005a0 0x7ff064069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff068fcd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064101790 0x7ff06419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff068fcd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff064102150 0x7ff06419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff068fcd640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06410c230 con 0x7ff064101790 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff068fcd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06410c0b0 con 0x7ff064102150 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff068fcd640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06410c3b0 con 0x7ff0641005a0 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff05bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064101790 0x7ff06419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff06259c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0641005a0 0x7ff064069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff05bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064101790 0x7ff06419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54536/0 (socket says 172.21.15.106:54536) 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.330+0000 7ff05bfff640 1 -- 172.21.15.106:0/1726404087 learned_addr learned my addr 172.21.15.106:0/1726404087 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff062d9d640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff064102150 0x7ff06419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff05bfff640 1 -- 172.21.15.106:0/1726404087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0641005a0 msgr2=0x7ff064069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff05bfff640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0641005a0 0x7ff064069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff05bfff640 1 -- 172.21.15.106:0/1726404087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff064102150 msgr2=0x7ff06419ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff05bfff640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff064102150 0x7ff06419ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff05bfff640 1 -- 172.21.15.106:0/1726404087 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06419f590 con 0x7ff064101790 2024-03-28T02:10:59.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff05bfff640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064101790 0x7ff06419c690 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00e9f0 tx=0x7ff04c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:59.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff05b7fe640 1 -- 172.21.15.106:0/1726404087 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04c00cd80 con 0x7ff064101790 2024-03-28T02:10:59.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff05b7fe640 1 -- 172.21.15.106:0/1726404087 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff04c010910 con 0x7ff064101790 2024-03-28T02:10:59.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06419f880 con 0x7ff064101790 2024-03-28T02:10:59.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff05b7fe640 1 -- 172.21.15.106:0/1726404087 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04c018990 con 0x7ff064101790 2024-03-28T02:10:59.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.331+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06419fe20 con 0x7ff064101790 2024-03-28T02:10:59.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.332+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0641078c0 con 0x7ff064101790 2024-03-28T02:10:59.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.333+0000 7ff05b7fe640 1 -- 172.21.15.106:0/1726404087 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff04c010ac0 con 0x7ff064101790 2024-03-28T02:10:59.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.334+0000 7ff05b7fe640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff0500a2880 0x7ff0500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.334+0000 7ff06259c640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff0500a2880 0x7ff0500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.334+0000 7ff05b7fe640 1 -- 172.21.15.106:0/1726404087 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff04c0cf490 con 0x7ff064101790 2024-03-28T02:10:59.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.334+0000 7ff06259c640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff0500a2880 0x7ff0500a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff0641015f0 tx=0x7ff054005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:59.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.337+0000 7ff05b7fe640 1 -- 172.21.15.106:0/1726404087 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff04c09bb40 con 0x7ff064101790 2024-03-28T02:10:59.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.517+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff06402d2c0 con 0x7ff064101790 2024-03-28T02:10:59.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.517+0000 7ff05b7fe640 1 -- 172.21.15.106:0/1726404087 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff04c09fbe0 con 0x7ff064101790 2024-03-28T02:10:59.510 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 64 2024-03-28T02:10:59.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff0500a2880 msgr2=0x7ff0500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff0500a2880 0x7ff0500a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff0641015f0 tx=0x7ff054005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064101790 msgr2=0x7ff06419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064101790 0x7ff06419c690 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00e9f0 tx=0x7ff04c00eeb0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 shutdown_connections 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff0500a2880 0x7ff0500a4d40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff064102150 0x7ff06419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff064101790 0x7ff06419c690 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 --2- 172.21.15.106:0/1726404087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0641005a0 0x7ff064069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 >> 172.21.15.106:0/1726404087 conn(0x7ff0640fbb30 msgr2=0x7ff064104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 shutdown_connections 2024-03-28T02:10:59.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.519+0000 7ff068fcd640 1 -- 172.21.15.106:0/1726404087 wait complete. 2024-03-28T02:10:59.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:59 smithi106 ceph-mon[31965]: pgmap v361: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T02:10:59.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:59 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3325087637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:59.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:59 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2113733214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:10:59.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:10:59 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4252798309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:10:59.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T02:10:59.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 -- 172.21.15.106:0/2633929937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa764102190 msgr2=0x7fa764108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 --2- 172.21.15.106:0/2633929937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa764102190 0x7fa764108a20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009ca0 tx=0x7fa75c02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 -- 172.21.15.106:0/2633929937 shutdown_connections 2024-03-28T02:10:59.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 --2- 172.21.15.106:0/2633929937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa764102190 0x7fa764108a20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 --2- 172.21.15.106:0/2633929937 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7641017d0 0x7fa764101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 --2- 172.21.15.106:0/2633929937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7641005e0 0x7fa764100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 -- 172.21.15.106:0/2633929937 >> 172.21.15.106:0/2633929937 conn(0x7fa7640fbb50 msgr2=0x7fa7640fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:59.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 -- 172.21.15.106:0/2633929937 shutdown_connections 2024-03-28T02:10:59.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 -- 172.21.15.106:0/2633929937 wait complete. 2024-03-28T02:10:59.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.635+0000 7fa769014640 1 Processor -- start 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa769014640 1 -- start start 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa769014640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7641005e0 0x7fa764196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa769014640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7641017d0 0x7fa764196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa769014640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa764102190 0x7fa76419dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa769014640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76410c240 con 0x7fa7641005e0 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa769014640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76410c0c0 con 0x7fa7641017d0 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa769014640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76410c3c0 con 0x7fa764102190 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa762d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7641005e0 0x7fa764196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa762d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7641005e0 0x7fa764196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54564/0 (socket says 172.21.15.106:54564) 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa762d9d640 1 -- 172.21.15.106:0/2903287806 learned_addr learned my addr 172.21.15.106:0/2903287806 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa75a59c640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7641017d0 0x7fa764196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa76359e640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa764102190 0x7fa76419dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa762d9d640 1 -- 172.21.15.106:0/2903287806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa764102190 msgr2=0x7fa76419dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa762d9d640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa764102190 0x7fa76419dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa762d9d640 1 -- 172.21.15.106:0/2903287806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7641017d0 msgr2=0x7fa764196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa762d9d640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7641017d0 0x7fa764196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa762d9d640 1 -- 172.21.15.106:0/2903287806 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76419e530 con 0x7fa7641005e0 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa762d9d640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7641005e0 0x7fa764196910 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa75400e9c0 tx=0x7fa75400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa760d99640 1 -- 172.21.15.106:0/2903287806 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75400cd80 con 0x7fa7641005e0 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.636+0000 7fa760d99640 1 -- 172.21.15.106:0/2903287806 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7540035f0 con 0x7fa7641005e0 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.637+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76419e820 con 0x7fa7641005e0 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.637+0000 7fa760d99640 1 -- 172.21.15.106:0/2903287806 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754010690 con 0x7fa7641005e0 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.637+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76419edc0 con 0x7fa7641005e0 2024-03-28T02:10:59.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.637+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa764107950 con 0x7fa7641005e0 2024-03-28T02:10:59.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.638+0000 7fa760d99640 1 -- 172.21.15.106:0/2903287806 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa7540037a0 con 0x7fa7641005e0 2024-03-28T02:10:59.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.639+0000 7fa760d99640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa7440a2880 0x7fa7440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.639+0000 7fa75a59c640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa7440a2880 0x7fa7440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.639+0000 7fa760d99640 1 -- 172.21.15.106:0/2903287806 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa7540cf130 con 0x7fa7641005e0 2024-03-28T02:10:59.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.640+0000 7fa75a59c640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa7440a2880 0x7fa7440a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa764197e20 tx=0x7fa74c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:59.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.643+0000 7fa760d99640 1 -- 172.21.15.106:0/2903287806 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa75409be30 con 0x7fa7641005e0 2024-03-28T02:10:59.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:59 smithi153 ceph-mon[36228]: pgmap v361: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T02:10:59.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:59 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3325087637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:59.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:59 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2113733214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:10:59.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:10:59 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4252798309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:10:59.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.818+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa7640622d0 con 0x7fa7641005e0 2024-03-28T02:10:59.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.818+0000 7fa760d99640 1 -- 172.21.15.106:0/2903287806 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa75409b730 con 0x7fa7641005e0 2024-03-28T02:10:59.811 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:10:59.813 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa7440a2880 msgr2=0x7fa7440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa7440a2880 0x7fa7440a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa764197e20 tx=0x7fa74c005f90 comp rx=0 tx=0).stop 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7641005e0 msgr2=0x7fa764196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7641005e0 0x7fa764196910 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa75400e9c0 tx=0x7fa75400ee80 comp rx=0 tx=0).stop 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 shutdown_connections 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa764102190 0x7fa76419dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa7440a2880 0x7fa7440a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7641017d0 0x7fa764196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 --2- 172.21.15.106:0/2903287806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7641005e0 0x7fa764196910 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 >> 172.21.15.106:0/2903287806 conn(0x7fa7640fbb50 msgr2=0x7fa764104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 shutdown_connections 2024-03-28T02:10:59.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.820+0000 7fa769014640 1 -- 172.21.15.106:0/2903287806 wait complete. 2024-03-28T02:10:59.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:10:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:59 smithi178 ceph-mon[36521]: pgmap v361: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T02:10:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:59 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3325087637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:10:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:59 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2113733214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:10:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:10:59 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4252798309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:10:59.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 -- 172.21.15.106:0/1395888776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec81016c0 msgr2=0x7faec8101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 --2- 172.21.15.106:0/1395888776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec81016c0 0x7faec8101ae0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7faebc009900 tx=0x7faebc02bc30 comp rx=0 tx=0).stop 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 -- 172.21.15.106:0/1395888776 shutdown_connections 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 --2- 172.21.15.106:0/1395888776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faec8103270 0x7faec8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 --2- 172.21.15.106:0/1395888776 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faec81028b0 0x7faec8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 --2- 172.21.15.106:0/1395888776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec81016c0 0x7faec8101ae0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 -- 172.21.15.106:0/1395888776 >> 172.21.15.106:0/1395888776 conn(0x7faec80fcc50 msgr2=0x7faec80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 -- 172.21.15.106:0/1395888776 shutdown_connections 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 -- 172.21.15.106:0/1395888776 wait complete. 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.933+0000 7faecc93b640 1 Processor -- start 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faecc93b640 1 -- start start 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faecc93b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faec81016c0 0x7faec81979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faecc93b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faec81028b0 0x7faec8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faecc93b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec8103270 0x7faec819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faecc93b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec810d310 con 0x7faec8103270 2024-03-28T02:10:59.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faecc93b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec810d190 con 0x7faec81016c0 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faecc93b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec810d490 con 0x7faec81028b0 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec6d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec8103270 0x7faec819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec6d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec8103270 0x7faec819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54578/0 (socket says 172.21.15.106:54578) 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec6d9d640 1 -- 172.21.15.106:0/2252008785 learned_addr learned my addr 172.21.15.106:0/2252008785 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec5d9b640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faec81028b0 0x7faec8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec659c640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faec81016c0 0x7faec81979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec6d9d640 1 -- 172.21.15.106:0/2252008785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faec81028b0 msgr2=0x7faec8197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec6d9d640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faec81028b0 0x7faec8197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec6d9d640 1 -- 172.21.15.106:0/2252008785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faec81016c0 msgr2=0x7faec81979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec6d9d640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faec81016c0 0x7faec81979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec6d9d640 1 -- 172.21.15.106:0/2252008785 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec819f5e0 con 0x7faec8103270 2024-03-28T02:10:59.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec5d9b640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faec81028b0 0x7faec8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:59.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec659c640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faec81016c0 0x7faec81979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:10:59.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faec6d9d640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec8103270 0x7faec819ef70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7faec000eb10 tx=0x7faec000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:59.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faeb77fe640 1 -- 172.21.15.106:0/2252008785 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0004b20 con 0x7faec8103270 2024-03-28T02:10:59.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faeb77fe640 1 -- 172.21.15.106:0/2252008785 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faec0004cc0 con 0x7faec8103270 2024-03-28T02:10:59.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faeb77fe640 1 -- 172.21.15.106:0/2252008785 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0010760 con 0x7faec8103270 2024-03-28T02:10:59.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.934+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec819f8d0 con 0x7faec8103270 2024-03-28T02:10:59.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.935+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec819fdd0 con 0x7faec8103270 2024-03-28T02:10:59.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.935+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec8108a00 con 0x7faec8103270 2024-03-28T02:10:59.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.936+0000 7faeb77fe640 1 -- 172.21.15.106:0/2252008785 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7faec0005020 con 0x7faec8103270 2024-03-28T02:10:59.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.937+0000 7faeb77fe640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7faeb00a2800 0x7faeb00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:10:59.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.937+0000 7faec659c640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7faeb00a2800 0x7faeb00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:10:59.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.937+0000 7faeb77fe640 1 -- 172.21.15.106:0/2252008785 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7faec00cf370 con 0x7faec8103270 2024-03-28T02:10:59.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.937+0000 7faec659c640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7faeb00a2800 0x7faeb00a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faebc000c00 tx=0x7faebc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:10:59.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:10:59.941+0000 7faeb77fe640 1 -- 172.21.15.106:0/2252008785 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faec009c070 con 0x7faec8103270 2024-03-28T02:11:00.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.153+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faec802d070 con 0x7faec8103270 2024-03-28T02:11:00.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.153+0000 7faeb77fe640 1 -- 172.21.15.106:0/2252008785 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11474 (secure 0 0 0) 0x7faec009b970 con 0x7faec8103270 2024-03-28T02:11:00.146 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:11:00.147 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:09:11.288485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24499,"mds_2":24460,"mds_3":24446,"mds_4":24415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14625":{"gid":14625,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14637":{"gid":14637,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24446":{"gid":24446,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24415,24499,24509,24446,24460]},"id":1}]} 2024-03-28T02:11:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 28 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7faeb00a2800 msgr2=0x7faeb00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7faeb00a2800 0x7faeb00a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faebc000c00 tx=0x7faebc005fd0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec8103270 msgr2=0x7faec819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec8103270 0x7faec819ef70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7faec000eb10 tx=0x7faec000c2d0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 shutdown_connections 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faec8103270 0x7faec819ef70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faec81028b0 0x7faec8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7faeb00a2800 0x7faeb00a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 --2- 172.21.15.106:0/2252008785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faec81016c0 0x7faec81979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 >> 172.21.15.106:0/2252008785 conn(0x7faec80fcc50 msgr2=0x7faec8105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 shutdown_connections 2024-03-28T02:11:00.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.155+0000 7faecc93b640 1 -- 172.21.15.106:0/2252008785 wait complete. 2024-03-28T02:11:00.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:11:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.268+0000 7f46cb1f4640 1 -- 172.21.15.106:0/788783189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 msgr2=0x7f46c4108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.268+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/788783189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 0x7f46c4108a20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009ca0 tx=0x7f46bc02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.268+0000 7f46cb1f4640 1 -- 172.21.15.106:0/788783189 shutdown_connections 2024-03-28T02:11:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.268+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/788783189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 0x7f46c4108a20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.268+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/788783189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46c41017d0 0x7f46c4101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.268+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/788783189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46c41005e0 0x7f46c4100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.268+0000 7f46cb1f4640 1 -- 172.21.15.106:0/788783189 >> 172.21.15.106:0/788783189 conn(0x7f46c40fbb50 msgr2=0x7f46c40fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.268+0000 7f46cb1f4640 1 -- 172.21.15.106:0/788783189 shutdown_connections 2024-03-28T02:11:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.268+0000 7f46cb1f4640 1 -- 172.21.15.106:0/788783189 wait complete. 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46cb1f4640 1 Processor -- start 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46cb1f4640 1 -- start start 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46cb1f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46c41005e0 0x7f46c41979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46cb1f4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46c41017d0 0x7f46c4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46cb1f4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 0x7f46c419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46cb1f4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c410c260 con 0x7f46c4102190 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46cb1f4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c410c0e0 con 0x7f46c41005e0 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46cb1f4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c410c3e0 con 0x7f46c41017d0 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c9791640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 0x7f46c419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c9791640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 0x7f46c419ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54606/0 (socket says 172.21.15.106:54606) 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c9791640 1 -- 172.21.15.106:0/3846831023 learned_addr learned my addr 172.21.15.106:0/3846831023 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c8f90640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46c41005e0 0x7f46c41979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c3fff640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46c41017d0 0x7f46c4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c9791640 1 -- 172.21.15.106:0/3846831023 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46c41017d0 msgr2=0x7f46c4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c9791640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46c41017d0 0x7f46c4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c9791640 1 -- 172.21.15.106:0/3846831023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46c41005e0 msgr2=0x7f46c41979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c9791640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46c41005e0 0x7f46c41979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.269+0000 7f46c9791640 1 -- 172.21.15.106:0/3846831023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c419f5c0 con 0x7f46c4102190 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.270+0000 7f46c9791640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 0x7f46c419ef50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f46bc02c630 tx=0x7f46bc032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.270+0000 7f46c1ffb640 1 -- 172.21.15.106:0/3846831023 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc041020 con 0x7f46c4102190 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.270+0000 7f46c1ffb640 1 -- 172.21.15.106:0/3846831023 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46bc034cd0 con 0x7f46c4102190 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.270+0000 7f46c1ffb640 1 -- 172.21.15.106:0/3846831023 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc039040 con 0x7f46c4102190 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.270+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c419f850 con 0x7f46c4102190 2024-03-28T02:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.270+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c419fd90 con 0x7f46c4102190 2024-03-28T02:11:00.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.271+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c41a0050 con 0x7f46c4102190 2024-03-28T02:11:00.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.272+0000 7f46c1ffb640 1 -- 172.21.15.106:0/3846831023 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f46bc044020 con 0x7f46c4102190 2024-03-28T02:11:00.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.272+0000 7f46c1ffb640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46b40a2880 0x7f46b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.273+0000 7f46c1ffb640 1 -- 172.21.15.106:0/3846831023 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f46bc0f0bd0 con 0x7f46c4102190 2024-03-28T02:11:00.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.273+0000 7f46c8f90640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46b40a2880 0x7f46b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.273+0000 7f46c8f90640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46b40a2880 0x7f46b40a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f46c4101630 tx=0x7f46b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:00.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.277+0000 7f46c1ffb640 1 -- 172.21.15.106:0/3846831023 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f46bc04b020 con 0x7f46c4102190 2024-03-28T02:11:00.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.454+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46c41a0740 con 0x7f46c4102190 2024-03-28T02:11:00.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.456+0000 7f46c1ffb640 1 -- 172.21.15.106:0/3846831023 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24845 (secure 0 0 0) 0x7f46bc0bd960 con 0x7f46c4102190 2024-03-28T02:11:00.449 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:11:00.449 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":93,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:10:43.590748+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:11:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46b40a2880 msgr2=0x7f46b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46b40a2880 0x7f46b40a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f46c4101630 tx=0x7f46b8005f90 comp rx=0 tx=0).stop 2024-03-28T02:11:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 msgr2=0x7f46c419ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 0x7f46c419ef50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f46bc02c630 tx=0x7f46bc032ea0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 shutdown_connections 2024-03-28T02:11:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46c4102190 0x7f46c419ef50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46c41017d0 0x7f46c4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f46b40a2880 0x7f46b40a4d40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 --2- 172.21.15.106:0/3846831023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46c41005e0 0x7f46c41979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 >> 172.21.15.106:0/3846831023 conn(0x7f46c40fbb50 msgr2=0x7f46c40fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 shutdown_connections 2024-03-28T02:11:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.458+0000 7f46cb1f4640 1 -- 172.21.15.106:0/3846831023 wait complete. 2024-03-28T02:11:00.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T02:08:46.952662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T02:08:47.976273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T02:08:51.703592+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T02:11:00.464 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 -- 172.21.15.106:0/1298538864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480102150 msgr2=0x7f24801089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 --2- 172.21.15.106:0/1298538864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480102150 0x7f24801089e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2478009ca0 tx=0x7f247802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 -- 172.21.15.106:0/1298538864 shutdown_connections 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 --2- 172.21.15.106:0/1298538864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480102150 0x7f24801089e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 --2- 172.21.15.106:0/1298538864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2480101790 0x7f2480101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 --2- 172.21.15.106:0/1298538864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24801005a0 0x7f24801009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 -- 172.21.15.106:0/1298538864 >> 172.21.15.106:0/1298538864 conn(0x7f24800fbb30 msgr2=0x7f24800fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 -- 172.21.15.106:0/1298538864 shutdown_connections 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 -- 172.21.15.106:0/1298538864 wait complete. 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 Processor -- start 2024-03-28T02:11:00.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.575+0000 7f248635b640 1 -- start start 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f248635b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24801005a0 0x7f24801979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f248635b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2480101790 0x7f2480197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f248635b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480102150 0x7f248019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f248635b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248010c230 con 0x7f2480102150 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f248635b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248010c0b0 con 0x7f24801005a0 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f248635b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248010c3b0 con 0x7f2480101790 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f24848f8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480102150 0x7f248019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f247ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24801005a0 0x7f24801979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f247ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24801005a0 0x7f24801979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:58300/0 (socket says 172.21.15.106:58300) 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f247ffff640 1 -- 172.21.15.106:0/3062859679 learned_addr learned my addr 172.21.15.106:0/3062859679 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f247f7fe640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2480101790 0x7f2480197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f24848f8640 1 -- 172.21.15.106:0/3062859679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2480101790 msgr2=0x7f2480197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f24848f8640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2480101790 0x7f2480197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f24848f8640 1 -- 172.21.15.106:0/3062859679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24801005a0 msgr2=0x7f24801979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f24848f8640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24801005a0 0x7f24801979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f24848f8640 1 -- 172.21.15.106:0/3062859679 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248019f5c0 con 0x7f2480102150 2024-03-28T02:11:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f247ffff640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24801005a0 0x7f24801979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f24848f8640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480102150 0x7f248019ef50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f247802c630 tx=0x7f2478032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f247d7fa640 1 -- 172.21.15.106:0/3062859679 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478041020 con 0x7f2480102150 2024-03-28T02:11:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f247d7fa640 1 -- 172.21.15.106:0/3062859679 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2478034cd0 con 0x7f2480102150 2024-03-28T02:11:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248019f850 con 0x7f2480102150 2024-03-28T02:11:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f247d7fa640 1 -- 172.21.15.106:0/3062859679 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478039040 con 0x7f2480102150 2024-03-28T02:11:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.576+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248019fd10 con 0x7f2480102150 2024-03-28T02:11:00.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.577+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248019ff70 con 0x7f2480102150 2024-03-28T02:11:00.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.578+0000 7f247d7fa640 1 -- 172.21.15.106:0/3062859679 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2478044020 con 0x7f2480102150 2024-03-28T02:11:00.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.579+0000 7f247d7fa640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f24680a2800 0x7f24680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.579+0000 7f247ffff640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f24680a2800 0x7f24680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.579+0000 7f247d7fa640 1 -- 172.21.15.106:0/3062859679 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f24780f0bb0 con 0x7f2480102150 2024-03-28T02:11:00.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.580+0000 7f247ffff640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f24680a2800 0x7f24680a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f24700058c0 tx=0x7f2470005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:00.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.584+0000 7f247d7fa640 1 -- 172.21.15.106:0/3062859679 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f247804b020 con 0x7f2480102150 2024-03-28T02:11:00.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:00 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1726404087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:11:00.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:00 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2903287806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:11:00.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:00 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2252008785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:11:00.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.758+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24801a0300 con 0x7f2480102150 2024-03-28T02:11:00.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.758+0000 7f247d7fa640 1 -- 172.21.15.106:0/3062859679 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24780bd940 con 0x7f2480102150 2024-03-28T02:11:00.752 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:11:00.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f24680a2800 msgr2=0x7f24680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f24680a2800 0x7f24680a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f24700058c0 tx=0x7f2470005f90 comp rx=0 tx=0).stop 2024-03-28T02:11:00.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480102150 msgr2=0x7f248019ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480102150 0x7f248019ef50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f247802c630 tx=0x7f2478032f20 comp rx=0 tx=0).stop 2024-03-28T02:11:00.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 shutdown_connections 2024-03-28T02:11:00.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480102150 0x7f248019ef50 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2480101790 0x7f2480197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f24680a2800 0x7f24680a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 --2- 172.21.15.106:0/3062859679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24801005a0 0x7f24801979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 >> 172.21.15.106:0/3062859679 conn(0x7f24800fbb30 msgr2=0x7f24800fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:00.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 shutdown_connections 2024-03-28T02:11:00.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.761+0000 7f248635b640 1 -- 172.21.15.106:0/3062859679 wait complete. 2024-03-28T02:11:00.764 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:00 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1726404087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:11:00.764 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:00 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2903287806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:11:00.764 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:00 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2252008785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:11:00.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 -- 172.21.15.106:0/3109086878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1016b0 msgr2=0x7f8b0c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 --2- 172.21.15.106:0/3109086878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1016b0 0x7f8b0c101ad0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009900 tx=0x7f8b0002bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 -- 172.21.15.106:0/3109086878 shutdown_connections 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 --2- 172.21.15.106:0/3109086878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b0c103260 0x7f8b0c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 --2- 172.21.15.106:0/3109086878 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b0c1028a0 0x7f8b0c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 --2- 172.21.15.106:0/3109086878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1016b0 0x7f8b0c101ad0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 -- 172.21.15.106:0/3109086878 >> 172.21.15.106:0/3109086878 conn(0x7f8b0c0fcc40 msgr2=0x7f8b0c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 -- 172.21.15.106:0/3109086878 shutdown_connections 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 -- 172.21.15.106:0/3109086878 wait complete. 2024-03-28T02:11:00.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 Processor -- start 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.877+0000 7f8b130a1640 1 -- start start 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b130a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b0c1016b0 0x7f8b0c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b130a1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1028a0 0x7f8b0c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b130a1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b0c103260 0x7f8b0c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b130a1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c10d300 con 0x7f8b0c1028a0 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b130a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c10d180 con 0x7f8b0c1016b0 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b130a1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c10d480 con 0x7f8b0c103260 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b0bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1028a0 0x7f8b0c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b0bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1028a0 0x7f8b0c19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54642/0 (socket says 172.21.15.106:54642) 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b0bfff640 1 -- 172.21.15.106:0/249701030 learned_addr learned my addr 172.21.15.106:0/249701030 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b10e3d640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b0c1016b0 0x7f8b0c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b1163e640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b0c103260 0x7f8b0c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b0bfff640 1 -- 172.21.15.106:0/249701030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b0c103260 msgr2=0x7f8b0c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b0bfff640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b0c103260 0x7f8b0c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b0bfff640 1 -- 172.21.15.106:0/249701030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b0c1016b0 msgr2=0x7f8b0c199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b0bfff640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b0c1016b0 0x7f8b0c199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b0bfff640 1 -- 172.21.15.106:0/249701030 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c1a1840 con 0x7f8b0c1028a0 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b10e3d640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b0c1016b0 0x7f8b0c199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b0bfff640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1028a0 0x7f8b0c19a170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8af800e9f0 tx=0x7f8af800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b09ffb640 1 -- 172.21.15.106:0/249701030 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af800cd80 con 0x7f8b0c1028a0 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0c1a1b30 con 0x7f8b0c1028a0 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b09ffb640 1 -- 172.21.15.106:0/249701030 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8af8010910 con 0x7f8b0c1028a0 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.878+0000 7f8b09ffb640 1 -- 172.21.15.106:0/249701030 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af80188f0 con 0x7f8b0c1028a0 2024-03-28T02:11:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.879+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0c1a2050 con 0x7f8b0c1028a0 2024-03-28T02:11:00.875 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:00 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1726404087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:11:00.875 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:00 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2903287806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:11:00.875 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:00 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2252008785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:11:00.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.880+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c1a2220 con 0x7f8b0c1028a0 2024-03-28T02:11:00.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.881+0000 7f8b09ffb640 1 -- 172.21.15.106:0/249701030 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8af8010ac0 con 0x7f8b0c1028a0 2024-03-28T02:11:00.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.881+0000 7f8b09ffb640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:00.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.881+0000 7f8b09ffb640 1 -- 172.21.15.106:0/249701030 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8af80cf240 con 0x7f8b0c1028a0 2024-03-28T02:11:00.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.881+0000 7f8b10e3d640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:00.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.882+0000 7f8b10e3d640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8b00000c00 tx=0x7f8b00005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:00.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:00.885+0000 7f8b09ffb640 1 -- 172.21.15.106:0/249701030 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8af809b8f0 con 0x7f8b0c1028a0 2024-03-28T02:11:01.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.058+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8b0c1a23f0 con 0x7f8b0c1028a0 2024-03-28T02:11:01.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.058+0000 7f8b09ffb640 1 -- 172.21.15.106:0/249701030 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8af809f990 con 0x7f8b0c1028a0 2024-03-28T02:11:01.052 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:11:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8afc0a2800 msgr2=0x7f8afc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8b00000c00 tx=0x7f8b00005fd0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1028a0 msgr2=0x7f8b0c19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1028a0 0x7f8b0c19a170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8af800e9f0 tx=0x7f8af800eeb0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 shutdown_connections 2024-03-28T02:11:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b0c103260 0x7f8b0c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b0c1028a0 0x7f8b0c19a170 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 --2- 172.21.15.106:0/249701030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b0c1016b0 0x7f8b0c199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 >> 172.21.15.106:0/249701030 conn(0x7f8b0c0fcc40 msgr2=0x7f8b0c108070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:01.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 shutdown_connections 2024-03-28T02:11:01.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.061+0000 7f8b130a1640 1 -- 172.21.15.106:0/249701030 wait complete. 2024-03-28T02:11:01.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T02:11:01.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.177+0000 7f17b78f1640 1 -- 172.21.15.106:0/1580004642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17b01017a0 msgr2=0x7f17b0101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.177+0000 7f17b78f1640 1 --2- 172.21.15.106:0/1580004642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17b01017a0 0x7f17b0101c20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009900 tx=0x7f17a002bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.177+0000 7f17b78f1640 1 -- 172.21.15.106:0/1580004642 shutdown_connections 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.177+0000 7f17b78f1640 1 --2- 172.21.15.106:0/1580004642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b0102160 0x7f17b01089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.177+0000 7f17b78f1640 1 --2- 172.21.15.106:0/1580004642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17b01017a0 0x7f17b0101c20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.177+0000 7f17b78f1640 1 --2- 172.21.15.106:0/1580004642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b01005b0 0x7f17b01009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.177+0000 7f17b78f1640 1 -- 172.21.15.106:0/1580004642 >> 172.21.15.106:0/1580004642 conn(0x7f17b00fbb40 msgr2=0x7f17b00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.177+0000 7f17b78f1640 1 -- 172.21.15.106:0/1580004642 shutdown_connections 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.177+0000 7f17b78f1640 1 -- 172.21.15.106:0/1580004642 wait complete. 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b78f1640 1 Processor -- start 2024-03-28T02:11:01.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b78f1640 1 -- start start 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b78f1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b01005b0 0x7f17b0197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b78f1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b01017a0 0x7f17b0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b78f1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17b0102160 0x7f17b019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b78f1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b010c210 con 0x7f17b01005b0 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b78f1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b010c090 con 0x7f17b01017a0 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b78f1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b010c390 con 0x7f17b0102160 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b568d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b01005b0 0x7f17b0197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b568d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b01005b0 0x7f17b0197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54654/0 (socket says 172.21.15.106:54654) 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b568d640 1 -- 172.21.15.106:0/2168064315 learned_addr learned my addr 172.21.15.106:0/2168064315 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b5e8e640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17b0102160 0x7f17b019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b4e8c640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b01017a0 0x7f17b0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b568d640 1 -- 172.21.15.106:0/2168064315 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17b0102160 msgr2=0x7f17b019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b568d640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17b0102160 0x7f17b019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b568d640 1 -- 172.21.15.106:0/2168064315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b01017a0 msgr2=0x7f17b0197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b568d640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b01017a0 0x7f17b0197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.178+0000 7f17b568d640 1 -- 172.21.15.106:0/2168064315 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b019f5c0 con 0x7f17b01005b0 2024-03-28T02:11:01.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.179+0000 7f17b568d640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b01005b0 0x7f17b0197a10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00e9c0 tx=0x7f17ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:01.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.179+0000 7f17a67fc640 1 -- 172.21.15.106:0/2168064315 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac00cd80 con 0x7f17b01005b0 2024-03-28T02:11:01.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.179+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b019f8b0 con 0x7f17b01005b0 2024-03-28T02:11:01.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.179+0000 7f17a67fc640 1 -- 172.21.15.106:0/2168064315 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17ac0035f0 con 0x7f17b01005b0 2024-03-28T02:11:01.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.179+0000 7f17a67fc640 1 -- 172.21.15.106:0/2168064315 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac010690 con 0x7f17b01005b0 2024-03-28T02:11:01.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.179+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b019fe20 con 0x7f17b01005b0 2024-03-28T02:11:01.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.180+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b01a00b0 con 0x7f17b01005b0 2024-03-28T02:11:01.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.181+0000 7f17a67fc640 1 -- 172.21.15.106:0/2168064315 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f17ac0037a0 con 0x7f17b01005b0 2024-03-28T02:11:01.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.181+0000 7f17a67fc640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f179c0a2800 0x7f179c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.181+0000 7f17b4e8c640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f179c0a2800 0x7f179c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.181+0000 7f17a67fc640 1 -- 172.21.15.106:0/2168064315 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f17ac0cf1b0 con 0x7f17b01005b0 2024-03-28T02:11:01.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.182+0000 7f17b4e8c640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f179c0a2800 0x7f179c0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f17a002c7b0 tx=0x7f17a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:01.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.186+0000 7f17a67fc640 1 -- 172.21.15.106:0/2168064315 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f17ac09beb0 con 0x7f17b01005b0 2024-03-28T02:11:01.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.361+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f17b01a0280 con 0x7f17b01005b0 2024-03-28T02:11:01.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.361+0000 7f17a67fc640 1 -- 172.21.15.106:0/2168064315 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f17ac09b7b0 con 0x7f17b01005b0 2024-03-28T02:11:01.355 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 64 2024-03-28T02:11:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.363+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f179c0a2800 msgr2=0x7f179c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.363+0000 7f17b78f1640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f179c0a2800 0x7f179c0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f17a002c7b0 tx=0x7f17a0005fd0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.363+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b01005b0 msgr2=0x7f17b0197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.363+0000 7f17b78f1640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b01005b0 0x7f17b0197a10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00e9c0 tx=0x7f17ac00ee80 comp rx=0 tx=0).stop 2024-03-28T02:11:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.363+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 shutdown_connections 2024-03-28T02:11:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.363+0000 7f17b78f1640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17b0102160 0x7f17b019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.364+0000 7f17b78f1640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f179c0a2800 0x7f179c0a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.364+0000 7f17b78f1640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b01017a0 0x7f17b0197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.364+0000 7f17b78f1640 1 --2- 172.21.15.106:0/2168064315 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b01005b0 0x7f17b0197a10 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.364+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 >> 172.21.15.106:0/2168064315 conn(0x7f17b00fbb40 msgr2=0x7f17b00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:01.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.364+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 shutdown_connections 2024-03-28T02:11:01.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.364+0000 7f17b78f1640 1 -- 172.21.15.106:0/2168064315 wait complete. 2024-03-28T02:11:01.368 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.482+0000 7fb556897640 1 -- 172.21.15.106:0/21555291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb550103260 msgr2=0x7fb550109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.482+0000 7fb556897640 1 --2- 172.21.15.106:0/21555291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb550103260 0x7fb550109af0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb548009ca0 tx=0x7fb54802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.482+0000 7fb556897640 1 -- 172.21.15.106:0/21555291 shutdown_connections 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.482+0000 7fb556897640 1 --2- 172.21.15.106:0/21555291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb550103260 0x7fb550109af0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.482+0000 7fb556897640 1 --2- 172.21.15.106:0/21555291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5501028a0 0x7fb550102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.482+0000 7fb556897640 1 --2- 172.21.15.106:0/21555291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5501016b0 0x7fb550101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.482+0000 7fb556897640 1 -- 172.21.15.106:0/21555291 >> 172.21.15.106:0/21555291 conn(0x7fb5500fcc40 msgr2=0x7fb5500ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.482+0000 7fb556897640 1 -- 172.21.15.106:0/21555291 shutdown_connections 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.482+0000 7fb556897640 1 -- 172.21.15.106:0/21555291 wait complete. 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb556897640 1 Processor -- start 2024-03-28T02:11:01.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb556897640 1 -- start start 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb556897640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5501016b0 0x7fb550197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb556897640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5501028a0 0x7fb550197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb556897640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb550103260 0x7fb55019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb556897640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55010d330 con 0x7fb5501016b0 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb556897640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55010d1b0 con 0x7fb5501028a0 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb556897640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55010d4b0 con 0x7fb550103260 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb54ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5501016b0 0x7fb550197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb54ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5501016b0 0x7fb550197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54686/0 (socket says 172.21.15.106:54686) 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb54ffff640 1 -- 172.21.15.106:0/1719307200 learned_addr learned my addr 172.21.15.106:0/1719307200 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb54f7fe640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5501028a0 0x7fb550197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb554e34640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb550103260 0x7fb55019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb54ffff640 1 -- 172.21.15.106:0/1719307200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb550103260 msgr2=0x7fb55019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb54ffff640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb550103260 0x7fb55019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb54ffff640 1 -- 172.21.15.106:0/1719307200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5501028a0 msgr2=0x7fb550197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb54ffff640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5501028a0 0x7fb550197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.483+0000 7fb54ffff640 1 -- 172.21.15.106:0/1719307200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55019f570 con 0x7fb5501016b0 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.484+0000 7fb54ffff640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5501016b0 0x7fb550197950 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb54000e9c0 tx=0x7fb54000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.484+0000 7fb54d7fa640 1 -- 172.21.15.106:0/1719307200 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54000cd80 con 0x7fb5501016b0 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.484+0000 7fb54d7fa640 1 -- 172.21.15.106:0/1719307200 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5400035f0 con 0x7fb5501016b0 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.484+0000 7fb54d7fa640 1 -- 172.21.15.106:0/1719307200 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540010690 con 0x7fb5501016b0 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.484+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55019f860 con 0x7fb5501016b0 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.484+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55019fd60 con 0x7fb5501016b0 2024-03-28T02:11:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.485+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5501089d0 con 0x7fb5501016b0 2024-03-28T02:11:01.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.486+0000 7fb54d7fa640 1 -- 172.21.15.106:0/1719307200 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb5400037a0 con 0x7fb5501016b0 2024-03-28T02:11:01.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.486+0000 7fb54d7fa640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb5380a2800 0x7fb5380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.487+0000 7fb54f7fe640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb5380a2800 0x7fb5380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.487+0000 7fb54d7fa640 1 -- 172.21.15.106:0/1719307200 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb5400cf160 con 0x7fb5501016b0 2024-03-28T02:11:01.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.487+0000 7fb54f7fe640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb5380a2800 0x7fb5380a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb550198e60 tx=0x7fb544005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:01.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.491+0000 7fb54d7fa640 1 -- 172.21.15.106:0/1719307200 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb54009be60 con 0x7fb5501016b0 2024-03-28T02:11:01.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.670+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb5500622a0 con 0x7fb5501016b0 2024-03-28T02:11:01.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.670+0000 7fb54d7fa640 1 -- 172.21.15.106:0/1719307200 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb54009b760 con 0x7fb5501016b0 2024-03-28T02:11:01.663 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:11:01.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb5380a2800 msgr2=0x7fb5380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb5380a2800 0x7fb5380a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb550198e60 tx=0x7fb544005f90 comp rx=0 tx=0).stop 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5501016b0 msgr2=0x7fb550197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5501016b0 0x7fb550197950 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb54000e9c0 tx=0x7fb54000ee80 comp rx=0 tx=0).stop 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 shutdown_connections 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb550103260 0x7fb55019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fb5380a2800 0x7fb5380a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5501028a0 0x7fb550197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 --2- 172.21.15.106:0/1719307200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5501016b0 0x7fb550197950 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 >> 172.21.15.106:0/1719307200 conn(0x7fb5500fcc40 msgr2=0x7fb550105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 shutdown_connections 2024-03-28T02:11:01.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.672+0000 7fb556897640 1 -- 172.21.15.106:0/1719307200 wait complete. 2024-03-28T02:11:01.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:01 smithi106 ceph-mon[31965]: pgmap v362: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T02:11:01.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:01 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3846831023' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:11:01.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:01 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3062859679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:11:01.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:01 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/249701030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:11:01.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:01 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2168064315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:11:01.677 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:11:01.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:01 smithi153 ceph-mon[36228]: pgmap v362: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T02:11:01.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:01 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3846831023' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:11:01.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:01 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3062859679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:11:01.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:01 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/249701030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:11:01.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:01 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2168064315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:11:01.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 -- 172.21.15.106:0/165018391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8107460 msgr2=0x7f6be8109850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 --2- 172.21.15.106:0/165018391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8107460 0x7f6be8109850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009900 tx=0x7f6be402bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:01.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 -- 172.21.15.106:0/165018391 shutdown_connections 2024-03-28T02:11:01.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 --2- 172.21.15.106:0/165018391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8107460 0x7f6be8109850 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 --2- 172.21.15.106:0/165018391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6be8104b30 0x7f6be8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 --2- 172.21.15.106:0/165018391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be8102200 0x7f6be81045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 -- 172.21.15.106:0/165018391 >> 172.21.15.106:0/165018391 conn(0x7f6be80fbb30 msgr2=0x7f6be80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 -- 172.21.15.106:0/165018391 shutdown_connections 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 -- 172.21.15.106:0/165018391 wait complete. 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 Processor -- start 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 -- start start 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6be8102200 0x7f6be81979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.786+0000 7f6befb97640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8104b30 0x7f6be8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6befb97640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be8107460 0x7f6be819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6befb97640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be810d350 con 0x7f6be8104b30 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6befb97640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be810d1d0 con 0x7f6be8107460 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6befb97640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be810d4d0 con 0x7f6be8102200 2024-03-28T02:11:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed132640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8104b30 0x7f6be8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed132640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8104b30 0x7f6be8197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54696/0 (socket says 172.21.15.106:54696) 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed132640 1 -- 172.21.15.106:0/3600882944 learned_addr learned my addr 172.21.15.106:0/3600882944 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed933640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6be8102200 0x7f6be81979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bee134640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be8107460 0x7f6be819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed132640 1 -- 172.21.15.106:0/3600882944 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6be8102200 msgr2=0x7f6be81979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed132640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6be8102200 0x7f6be81979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed132640 1 -- 172.21.15.106:0/3600882944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be8107460 msgr2=0x7f6be819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed132640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be8107460 0x7f6be819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed132640 1 -- 172.21.15.106:0/3600882944 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be819f5c0 con 0x7f6be8104b30 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bed132640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8104b30 0x7f6be8197ee0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8005fd0 tx=0x7f6bd800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bdeffd640 1 -- 172.21.15.106:0/3600882944 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8017070 con 0x7f6be8104b30 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bdeffd640 1 -- 172.21.15.106:0/3600882944 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bd8009c30 con 0x7f6be8104b30 2024-03-28T02:11:01.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be819f8b0 con 0x7f6be8104b30 2024-03-28T02:11:01.784 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6bdeffd640 1 -- 172.21.15.106:0/3600882944 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8010460 con 0x7f6be8104b30 2024-03-28T02:11:01.784 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.787+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be819fe50 con 0x7f6be8104b30 2024-03-28T02:11:01.784 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.788+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be80fd620 con 0x7f6be8104b30 2024-03-28T02:11:01.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.789+0000 7f6bdeffd640 1 -- 172.21.15.106:0/3600882944 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6bd8003e70 con 0x7f6be8104b30 2024-03-28T02:11:01.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.790+0000 7f6bdeffd640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6bd40a2880 0x7f6bd40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:01.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.790+0000 7f6bed933640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6bd40a2880 0x7f6bd40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:01.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.790+0000 7f6bdeffd640 1 -- 172.21.15.106:0/3600882944 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6bd80cf0d0 con 0x7f6be8104b30 2024-03-28T02:11:01.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.790+0000 7f6bed933640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6bd40a2880 0x7f6bd40a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6be0005fd0 tx=0x7f6be0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:01.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:01.794+0000 7f6bdeffd640 1 -- 172.21.15.106:0/3600882944 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6bd8020030 con 0x7f6be8104b30 2024-03-28T02:11:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:01 smithi178 ceph-mon[36521]: pgmap v362: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T02:11:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:01 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3846831023' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:11:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:01 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3062859679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:11:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:01 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/249701030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:11:01.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:01 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2168064315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:11:01.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.006+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6be802d070 con 0x7f6be8104b30 2024-03-28T02:11:01.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.006+0000 7f6bdeffd640 1 -- 172.21.15.106:0/3600882944 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11474 (secure 0 0 0) 0x7f6bd809be50 con 0x7f6be8104b30 2024-03-28T02:11:02.000 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:11:02.000 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:09:11.288485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24499,"mds_2":24460,"mds_3":24446,"mds_4":24415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14625":{"gid":14625,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14637":{"gid":14637,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24446":{"gid":24446,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24415,24499,24509,24446,24460]},"id":1}]} 2024-03-28T02:11:02.000 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 28 2024-03-28T02:11:02.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6bd40a2880 msgr2=0x7f6bd40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6bd40a2880 0x7f6bd40a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6be0005fd0 tx=0x7f6be0009500 comp rx=0 tx=0).stop 2024-03-28T02:11:02.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8104b30 msgr2=0x7f6be8197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8104b30 0x7f6be8197ee0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8005fd0 tx=0x7f6bd800dde0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 shutdown_connections 2024-03-28T02:11:02.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f6bd40a2880 0x7f6bd40a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be8107460 0x7f6be819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6be8104b30 0x7f6be8197ee0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 --2- 172.21.15.106:0/3600882944 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6be8102200 0x7f6be81979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 >> 172.21.15.106:0/3600882944 conn(0x7f6be80fbb30 msgr2=0x7f6be8102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 shutdown_connections 2024-03-28T02:11:02.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.008+0000 7f6befb97640 1 -- 172.21.15.106:0/3600882944 wait complete. 2024-03-28T02:11:02.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:11:02.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 -- 172.21.15.106:0/825622430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 msgr2=0x7f0214101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 --2- 172.21.15.106:0/825622430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 0x7f0214101c40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0210009900 tx=0x7f021002bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:02.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 -- 172.21.15.106:0/825622430 shutdown_connections 2024-03-28T02:11:02.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 --2- 172.21.15.106:0/825622430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0214102180 0x7f0214108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 --2- 172.21.15.106:0/825622430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 0x7f0214101c40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 --2- 172.21.15.106:0/825622430 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02141005d0 0x7f02141009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 -- 172.21.15.106:0/825622430 >> 172.21.15.106:0/825622430 conn(0x7f02140fbb80 msgr2=0x7f02140fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 -- 172.21.15.106:0/825622430 shutdown_connections 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 -- 172.21.15.106:0/825622430 wait complete. 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.123+0000 7f021c13b640 1 Processor -- start 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f021c13b640 1 -- start start 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f021c13b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02141005d0 0x7f0214196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f021c13b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 0x7f0214196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f021c13b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0214102180 0x7f021419dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f021c13b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021410c250 con 0x7f02141017c0 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f021c13b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021410c0d0 con 0x7f02141005d0 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f021c13b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021410c3d0 con 0x7f0214102180 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f02196d6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 0x7f0214196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f0219ed7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02141005d0 0x7f0214196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f02196d6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 0x7f0214196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54704/0 (socket says 172.21.15.106:54704) 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f02196d6640 1 -- 172.21.15.106:0/3121517077 learned_addr learned my addr 172.21.15.106:0/3121517077 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f021a6d8640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0214102180 0x7f021419dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f02196d6640 1 -- 172.21.15.106:0/3121517077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0214102180 msgr2=0x7f021419dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f02196d6640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0214102180 0x7f021419dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f02196d6640 1 -- 172.21.15.106:0/3121517077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02141005d0 msgr2=0x7f0214196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f02196d6640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02141005d0 0x7f0214196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f02196d6640 1 -- 172.21.15.106:0/3121517077 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021419e530 con 0x7f02141017c0 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f0219ed7640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02141005d0 0x7f0214196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f02196d6640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 0x7f0214196e50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0210009230 tx=0x7f021002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f020affd640 1 -- 172.21.15.106:0/3121517077 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210041070 con 0x7f02141017c0 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f020affd640 1 -- 172.21.15.106:0/3121517077 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0210004450 con 0x7f02141017c0 2024-03-28T02:11:02.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f020affd640 1 -- 172.21.15.106:0/3121517077 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210003db0 con 0x7f02141017c0 2024-03-28T02:11:02.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.124+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021419e7c0 con 0x7f02141017c0 2024-03-28T02:11:02.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.125+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021419ec60 con 0x7f02141017c0 2024-03-28T02:11:02.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.125+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02141079c0 con 0x7f02141017c0 2024-03-28T02:11:02.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.127+0000 7f020affd640 1 -- 172.21.15.106:0/3121517077 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0210044070 con 0x7f02141017c0 2024-03-28T02:11:02.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.127+0000 7f020affd640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f02000a2800 0x7f02000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.127+0000 7f0219ed7640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f02000a2800 0x7f02000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.127+0000 7f020affd640 1 -- 172.21.15.106:0/3121517077 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f02100f0d40 con 0x7f02141017c0 2024-03-28T02:11:02.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.128+0000 7f0219ed7640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f02000a2800 0x7f02000a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0214101620 tx=0x7f0204005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.131+0000 7f020affd640 1 -- 172.21.15.106:0/3121517077 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f02100bda40 con 0x7f02141017c0 2024-03-28T02:11:02.301 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.308+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f021402d240 con 0x7f02141017c0 2024-03-28T02:11:02.302 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.309+0000 7f020affd640 1 -- 172.21.15.106:0/3121517077 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24845 (secure 0 0 0) 0x7f02100a3120 con 0x7f02141017c0 2024-03-28T02:11:02.303 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:11:02.303 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":93,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:10:43.590748+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:11:02.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f02000a2800 msgr2=0x7f02000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f02000a2800 0x7f02000a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0214101620 tx=0x7f0204005f90 comp rx=0 tx=0).stop 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 msgr2=0x7f0214196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 0x7f0214196e50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0210009230 tx=0x7f021002c650 comp rx=0 tx=0).stop 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 shutdown_connections 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0214102180 0x7f021419dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02141017c0 0x7f0214196e50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f02000a2800 0x7f02000a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 --2- 172.21.15.106:0/3121517077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02141005d0 0x7f0214196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 >> 172.21.15.106:0/3121517077 conn(0x7f02140fbb80 msgr2=0x7f0214104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 shutdown_connections 2024-03-28T02:11:02.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.312+0000 7f021c13b640 1 -- 172.21.15.106:0/3121517077 wait complete. 2024-03-28T02:11:02.317 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T02:08:48.782692+0000', 'modified': '2024-03-28T02:09:11.288485+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24509, 'mds_1': 24499, 'mds_2': 24460, 'mds_3': 24446, 'mds_4': 24415}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14625': {'gid': 14625, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6839/1515774363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6838', 'nonce': 1515774363}, {'type': 'v1', 'addr': '172.21.15.106:6839', 'nonce': 1515774363}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14637': {'gid': 14637, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6841/1766684450', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6840', 'nonce': 1766684450}, {'type': 'v1', 'addr': '172.21.15.106:6841', 'nonce': 1766684450}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24404': {'gid': 24404, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6833/564442660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6832', 'nonce': 564442660}, {'type': 'v1', 'addr': '172.21.15.153:6833', 'nonce': 564442660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24415': {'gid': 24415, 'name': 'l', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.178:6833/1496777187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6832', 'nonce': 1496777187}, {'type': 'v1', 'addr': '172.21.15.178:6833', 'nonce': 1496777187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24446': {'gid': 24446, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.153:6835/3469968496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 3469968496}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 3469968496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.178:6835/1402366472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6834', 'nonce': 1402366472}, {'type': 'v1', 'addr': '172.21.15.178:6835', 'nonce': 1402366472}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6837/3554528514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6836', 'nonce': 3554528514}, {'type': 'v1', 'addr': '172.21.15.178:6837', 'nonce': 3554528514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6837/1253460068', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 1253460068}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 1253460068}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.178:6839/937216083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6838', 'nonce': 937216083}, {'type': 'v1', 'addr': '172.21.15.178:6839', 'nonce': 937216083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6839/505906484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 505906484}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 505906484}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24509, 'qdb_cluster': [24415, 24499, 24509, 24446, 24460]} 2024-03-28T02:11:02.317 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T02:11:02.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-03-28T02:11:02.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.427+0000 7f1241509640 1 -- 172.21.15.106:0/1829946831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 msgr2=0x7f123c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.427+0000 7f1241509640 1 --2- 172.21.15.106:0/1829946831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 0x7f123c101ad0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f122c009900 tx=0x7f122c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:02.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.427+0000 7f1241509640 1 -- 172.21.15.106:0/1829946831 shutdown_connections 2024-03-28T02:11:02.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.427+0000 7f1241509640 1 --2- 172.21.15.106:0/1829946831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f123c103260 0x7f123c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.427+0000 7f1241509640 1 --2- 172.21.15.106:0/1829946831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f123c1028a0 0x7f123c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.427+0000 7f1241509640 1 --2- 172.21.15.106:0/1829946831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 0x7f123c101ad0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.427+0000 7f1241509640 1 -- 172.21.15.106:0/1829946831 >> 172.21.15.106:0/1829946831 conn(0x7f123c0fcc40 msgr2=0x7f123c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.427+0000 7f1241509640 1 -- 172.21.15.106:0/1829946831 shutdown_connections 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.427+0000 7f1241509640 1 -- 172.21.15.106:0/1829946831 wait complete. 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f1241509640 1 Processor -- start 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f1241509640 1 -- start start 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f1241509640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 0x7f123c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f1241509640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f123c1028a0 0x7f123c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f1241509640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f123c103260 0x7f123c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f1241509640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c10d300 con 0x7f123c1016b0 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f1241509640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c10d180 con 0x7f123c1028a0 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f1241509640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c10d480 con 0x7f123c103260 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f123affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 0x7f123c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f123affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 0x7f123c197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54722/0 (socket says 172.21.15.106:54722) 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f123affd640 1 -- 172.21.15.106:0/652626919 learned_addr learned my addr 172.21.15.106:0/652626919 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f12327fc640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f123c1028a0 0x7f123c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f123b7fe640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f123c103260 0x7f123c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f123affd640 1 -- 172.21.15.106:0/652626919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f123c103260 msgr2=0x7f123c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f123affd640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f123c103260 0x7f123c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f123affd640 1 -- 172.21.15.106:0/652626919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f123c1028a0 msgr2=0x7f123c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f123affd640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f123c1028a0 0x7f123c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.428+0000 7f123affd640 1 -- 172.21.15.106:0/652626919 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c19f620 con 0x7f123c1016b0 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.429+0000 7f123affd640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 0x7f123c197a00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f122c000c00 tx=0x7f122c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.429+0000 7f1238ff9640 1 -- 172.21.15.106:0/652626919 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c041070 con 0x7f123c1016b0 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.429+0000 7f1238ff9640 1 -- 172.21.15.106:0/652626919 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f122c004450 con 0x7f123c1016b0 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.429+0000 7f1241509640 1 -- 172.21.15.106:0/652626919 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123c19f8b0 con 0x7f123c1016b0 2024-03-28T02:11:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.429+0000 7f1238ff9640 1 -- 172.21.15.106:0/652626919 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c034440 con 0x7f123c1016b0 2024-03-28T02:11:02.425 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.429+0000 7f1241509640 1 -- 172.21.15.106:0/652626919 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123c19fda0 con 0x7f123c1016b0 2024-03-28T02:11:02.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.431+0000 7f1238ff9640 1 -- 172.21.15.106:0/652626919 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f122c044070 con 0x7f123c1016b0 2024-03-28T02:11:02.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.431+0000 7f12441d4740 1 Processor -- start 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.431+0000 7f12441d4740 1 -- start start 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.431+0000 7f12441d4740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 0x55f5f910db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.431+0000 7f12441d4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9112c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.431+0000 7f12441d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 0x55f5f9115730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.431+0000 7f12441d4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f9072c20 con 0x55f5f910e130 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12441d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f901ba40 con 0x55f5f91132a0 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12441d4740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f906ecc0 con 0x55f5f910d740 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f1238ff9640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f12280a2800 0x7f12280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f123affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9112c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f123b7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 0x55f5f910db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f123affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9112c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54730/0 (socket says 172.21.15.106:54730) 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f123affd640 1 -- 172.21.15.106:0/2491313622 learned_addr learned my addr 172.21.15.106:0/2491313622 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12327fc640 1 --2- 172.21.15.106:0/2491313622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 0x55f5f9115730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f123b7fe640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f12280a2800 0x7f12280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f1238ff9640 1 -- 172.21.15.106:0/652626919 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f122c0f08c0 con 0x7f123c1016b0 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12327fc640 1 -- 172.21.15.106:0/2491313622 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 msgr2=0x55f5f910db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12327fc640 1 --2- 172.21.15.106:0/2491313622 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 0x55f5f910db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12327fc640 1 -- 172.21.15.106:0/2491313622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 msgr2=0x55f5f9112c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12327fc640 1 --2- 172.21.15.106:0/2491313622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9112c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12327fc640 1 -- 172.21.15.106:0/2491313622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5f903df90 con 0x55f5f91132a0 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12327fc640 1 --2- 172.21.15.106:0/2491313622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 0x55f5f9115730 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1224009bf0 tx=0x7f1224011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d90ff3966845dc25 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f123b7fe640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f12280a2800 0x7f12280a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55f5f910e990 tx=0x7f1234009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12317fa640 1 -- 172.21.15.106:0/2491313622 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224004050 con 0x55f5f91132a0 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12317fa640 1 -- 172.21.15.106:0/2491313622 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1224005500 con 0x55f5f91132a0 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.432+0000 7f12317fa640 1 -- 172.21.15.106:0/2491313622 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224004e00 con 0x55f5f91132a0 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.433+0000 7f12441d4740 1 -- 172.21.15.106:0/2491313622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 msgr2=0x55f5f9115730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.433+0000 7f12441d4740 1 --2- 172.21.15.106:0/2491313622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 0x55f5f9115730 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1224009bf0 tx=0x7f1224011810 comp rx=0 tx=0).stop 2024-03-28T02:11:02.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.433+0000 7f12441d4740 1 -- 172.21.15.106:0/2491313622 shutdown_connections 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.433+0000 7f12441d4740 1 --2- 172.21.15.106:0/2491313622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 0x55f5f9115730 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.433+0000 7f12441d4740 1 --2- 172.21.15.106:0/2491313622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9112c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.433+0000 7f12441d4740 1 --2- 172.21.15.106:0/2491313622 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 0x55f5f910db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.433+0000 7f12441d4740 1 -- 172.21.15.106:0/2491313622 >> 172.21.15.106:0/2491313622 conn(0x55f5f90d5a60 msgr2=0x55f5f91090b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.433+0000 7f12441d4740 1 -- 172.21.15.106:0/2491313622 shutdown_connections 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.433+0000 7f12441d4740 1 -- 172.21.15.106:0/2491313622 wait complete. 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12441d4740 1 -- start start 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12441d4740 1 Processor -- start 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12441d4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910d740 0x55f5f9117bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12441d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910e130 0x55f5f9118100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12441d4740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 0x55f5f911f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12441d4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f9116250 con 0x55f5f910d740 2024-03-28T02:11:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12441d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f902ee90 con 0x55f5f910e130 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12441d4740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f91163d0 con 0x55f5f91132a0 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12441d4740 10 client.0 _ll_register_callbacks cb 0x55f5f8d86ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f123affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910d740 0x55f5f9117bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12327fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910e130 0x55f5f9118100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f123affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910d740 0x55f5f9117bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54738/0 (socket says 172.21.15.106:54738) 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f123affd640 1 -- 172.21.15.106:0/2089332699 learned_addr learned my addr 172.21.15.106:0/2089332699 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f123b7fe640 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 0x55f5f911f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12327fc640 1 -- 172.21.15.106:0/2089332699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 msgr2=0x55f5f911f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12327fc640 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 0x55f5f911f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12327fc640 1 -- 172.21.15.106:0/2089332699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910d740 msgr2=0x55f5f9117bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12327fc640 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910d740 0x55f5f9117bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12327fc640 1 -- 172.21.15.106:0/2089332699 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5f903b9d0 con 0x55f5f910e130 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f123affd640 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910d740 0x55f5f9117bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:02.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12327fc640 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910e130 0x55f5f9118100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1224000c00 tx=0x7f12240098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1114d9879b578ad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.434+0000 7f12227fc640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12227fc640 1 -- 172.21.15.106:0/2089332699 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224029070 con 0x55f5f910e130 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12227fc640 1 -- 172.21.15.106:0/2089332699 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1224004d80 con 0x55f5f910e130 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12441d4740 1 -- 172.21.15.106:0/2089332699 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1224005500 con 0x55f5f910e130 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12227fc640 1 -- 172.21.15.106:0/2089332699 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1224035050 con 0x55f5f910e130 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12441d4740 10 client.25286 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12441d4740 1 -- 172.21.15.106:0/2089332699 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f5f8d872d0 con 0x55f5f910e130 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12227fc640 1 -- 172.21.15.106:0/2089332699 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1224005500 con 0x55f5f910e130 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12227fc640 1 -- 172.21.15.106:0/2089332699 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f1224003d30 con 0x55f5f910e130 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12441d4740 10 client.25286 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12441d4740 10 client.25286 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12441d4740 20 client.25286 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12441d4740 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x55f5f91240d0 0x55f5f9123300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12441d4740 4 client.25286 mds_command: new command op to 24415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f12441d4740 1 -- 172.21.15.106:0/2089332699 --> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5f9080160 con 0x55f5f91240d0 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.435+0000 7f123affd640 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x55f5f91240d0 0x55f5f9123300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12227fc640 10 client.25286 ms_handle_connect on v2:172.21.15.178:6832/1496777187 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f123affd640 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x55f5f91240d0 0x55f5f9123300 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7d3eefaae8d79125 server_cookie=463e11ce110ede32 in_seq=0 out_seq=0 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12227fc640 1 -- 172.21.15.106:0/2089332699 <== mds.4 v2:172.21.15.178:6832/1496777187 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f5f9080160 con 0x55f5f91240d0 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12227fc640 10 client.25286 handle_command_reply: tid=0 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12441d4740 1 client.25286 shutdown 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12441d4740 1 -- 172.21.15.106:0/2089332699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910e130 msgr2=0x55f5f9118100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12441d4740 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910e130 0x55f5f9118100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1224000c00 tx=0x7f12240098d0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12441d4740 1 -- 172.21.15.106:0/2089332699 shutdown_connections 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12441d4740 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x55f5f91240d0 0x55f5f9123300 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12441d4740 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 0x55f5f911f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12441d4740 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910e130 0x55f5f9118100 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12441d4740 1 --2- 172.21.15.106:0/2089332699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910d740 0x55f5f9117bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12227fc640 0 client.25286 ms_handle_reset on v2:172.21.15.178:6832/1496777187 2024-03-28T02:11:02.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.436+0000 7f12441d4740 1 -- 172.21.15.106:0/2089332699 >> 172.21.15.106:0/2089332699 conn(0x55f5f90d5a60 msgr2=0x55f5f91082f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.437+0000 7f12441d4740 1 -- 172.21.15.106:0/2089332699 shutdown_connections 2024-03-28T02:11:02.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.437+0000 7f12441d4740 1 -- 172.21.15.106:0/2089332699 wait complete. 2024-03-28T02:11:02.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.437+0000 7f12441d4740 20 client.25286 trim_cache size 0 max 16384 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12441d4740 1 Processor -- start 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12441d4740 1 -- start start 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12441d4740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 0x55f5f9108bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12441d4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9109110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12441d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 0x55f5f910c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12441d4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f91238f0 con 0x55f5f910e130 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12441d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f9123f50 con 0x55f5f91132a0 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12441d4740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f91245b0 con 0x55f5f910d740 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12327fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9109110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12327fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9109110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54752/0 (socket says 172.21.15.106:54752) 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f12327fc640 1 -- 172.21.15.106:0/1036890109 learned_addr learned my addr 172.21.15.106:0/1036890109 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f123b7fe640 1 --2- 172.21.15.106:0/1036890109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 0x55f5f910c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.453+0000 7f123affd640 1 --2- 172.21.15.106:0/1036890109 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 0x55f5f9108bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f12327fc640 1 -- 172.21.15.106:0/1036890109 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 msgr2=0x55f5f9108bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f12327fc640 1 --2- 172.21.15.106:0/1036890109 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 0x55f5f9108bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f12327fc640 1 -- 172.21.15.106:0/1036890109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 msgr2=0x55f5f910c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f12327fc640 1 --2- 172.21.15.106:0/1036890109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 0x55f5f910c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f12327fc640 1 -- 172.21.15.106:0/1036890109 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5f8e516b0 con 0x55f5f910e130 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f12327fc640 1 --2- 172.21.15.106:0/1036890109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9109110 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1224005e00 tx=0x7f1224005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96753bfd7b9875a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f1230ff9640 1 -- 172.21.15.106:0/1036890109 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224029070 con 0x55f5f910e130 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f1230ff9640 1 -- 172.21.15.106:0/1036890109 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12240035f0 con 0x55f5f910e130 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f1230ff9640 1 -- 172.21.15.106:0/1036890109 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224019690 con 0x55f5f910e130 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f12441d4740 1 -- 172.21.15.106:0/1036890109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 msgr2=0x55f5f9109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.454+0000 7f12441d4740 1 --2- 172.21.15.106:0/1036890109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9109110 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1224005e00 tx=0x7f1224005f90 comp rx=0 tx=0).stop 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 -- 172.21.15.106:0/1036890109 shutdown_connections 2024-03-28T02:11:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 --2- 172.21.15.106:0/1036890109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f91132a0 0x55f5f910c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 --2- 172.21.15.106:0/1036890109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f9109110 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 --2- 172.21.15.106:0/1036890109 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f910d740 0x55f5f9108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 -- 172.21.15.106:0/1036890109 >> 172.21.15.106:0/1036890109 conn(0x55f5f90d5a60 msgr2=0x55f5f9117bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 -- 172.21.15.106:0/1036890109 shutdown_connections 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 -- 172.21.15.106:0/1036890109 wait complete. 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 -- start start 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 Processor -- start 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910d740 0x55f5f917b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f917b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 0x55f5f9182480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f9125270 con 0x55f5f910e130 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f9124c10 con 0x55f5f910d740 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f91258d0 con 0x55f5f91132a0 2024-03-28T02:11:02.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12441d4740 10 client.0 _ll_register_callbacks cb 0x7f122c0b8c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12327fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f917b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12327fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f917b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54762/0 (socket says 172.21.15.106:54762) 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12327fc640 1 -- 172.21.15.106:0/2391784639 learned_addr learned my addr 172.21.15.106:0/2391784639 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f123b7fe640 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 0x55f5f9182480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f123affd640 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910d740 0x55f5f917b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12327fc640 1 -- 172.21.15.106:0/2391784639 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 msgr2=0x55f5f9182480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12327fc640 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 0x55f5f9182480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12327fc640 1 -- 172.21.15.106:0/2391784639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910d740 msgr2=0x55f5f917b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12327fc640 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910d740 0x55f5f917b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.455+0000 7f12327fc640 1 -- 172.21.15.106:0/2391784639 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5f9182a70 con 0x55f5f910e130 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12327fc640 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f917b8a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1224009bf0 tx=0x7f12240039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a76e91cffab202 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12227fc640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12227fc640 1 -- 172.21.15.106:0/2391784639 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224032070 con 0x55f5f910e130 2024-03-28T02:11:02.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12227fc640 1 -- 172.21.15.106:0/2391784639 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f12240236f0 con 0x55f5f910e130 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12441d4740 1 -- 172.21.15.106:0/2391784639 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12240035f0 con 0x55f5f910e130 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12227fc640 1 -- 172.21.15.106:0/2391784639 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1224047050 con 0x55f5f910e130 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12441d4740 10 client.15444 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12441d4740 1 -- 172.21.15.106:0/2391784639 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f5f9183d70 con 0x55f5f910e130 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12227fc640 1 -- 172.21.15.106:0/2391784639 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12240035f0 con 0x55f5f910e130 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12227fc640 1 -- 172.21.15.106:0/2391784639 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f122402ead0 con 0x55f5f910e130 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12441d4740 10 client.15444 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12441d4740 10 client.15444 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12441d4740 20 client.15444 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12441d4740 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x55f5f9188020 0x55f5f9183fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12441d4740 4 client.15444 mds_command: new command op to 24415 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f12441d4740 1 -- 172.21.15.106:0/2391784639 --> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f5f907a4b0 con 0x55f5f9188020 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.456+0000 7f123affd640 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x55f5f9188020 0x55f5f9183fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12227fc640 10 client.15444 ms_handle_connect on v2:172.21.15.178:6832/1496777187 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f123affd640 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x55f5f9188020 0x55f5f9183fa0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7a1e86e1ebda68d6 server_cookie=af58323ed1378443 in_seq=0 out_seq=0 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12227fc640 1 -- 172.21.15.106:0/2391784639 <== mds.4 v2:172.21.15.178:6832/1496777187 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55f5f907a4b0 con 0x55f5f9188020 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12227fc640 10 client.15444 handle_command_reply: tid=0 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 client.15444 shutdown 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 -- 172.21.15.106:0/2391784639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 msgr2=0x55f5f917b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f917b8a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1224009bf0 tx=0x7f12240039d0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 -- 172.21.15.106:0/2391784639 shutdown_connections 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.178:6832/1496777187,v1:172.21.15.178:6833/1496777187] conn(0x55f5f9188020 0x55f5f9183fa0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f5f91132a0 0x55f5f9182480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12227fc640 0 client.15444 ms_handle_reset on v2:172.21.15.178:6832/1496777187 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5f910e130 0x55f5f917b8a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 --2- 172.21.15.106:0/2391784639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5f910d740 0x55f5f917b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 -- 172.21.15.106:0/2391784639 >> 172.21.15.106:0/2391784639 conn(0x55f5f90d5a60 msgr2=0x55f5f9108bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 -- 172.21.15.106:0/2391784639 shutdown_connections 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 1 -- 172.21.15.106:0/2391784639 wait complete. 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.457+0000 7f12441d4740 20 client.15444 trim_cache size 0 max 16384 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 4, 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24415, 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:11:02.458 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 112.188931356, 2024-03-28T02:11:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:11:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 93, 2024-03-28T02:11:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:11:02.459 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 171.411202454 2024-03-28T02:11:02.459 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 -- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f12280a2800 msgr2=0x7f12280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f12280a2800 0x7f12280a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55f5f910e990 tx=0x7f1234009500 comp rx=0 tx=0).stop 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 -- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 msgr2=0x7f123c197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 0x7f123c197a00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f122c000c00 tx=0x7f122c02c640 comp rx=0 tx=0).stop 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 -- 172.21.15.106:0/652626919 shutdown_connections 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f123c103260 0x7f123c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f12280a2800 0x7f12280a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f123c1028a0 0x7f123c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 --2- 172.21.15.106:0/652626919 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f123c1016b0 0x7f123c197a00 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 -- 172.21.15.106:0/652626919 >> 172.21.15.106:0/652626919 conn(0x7f123c0fcc40 msgr2=0x7f123c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 -- 172.21.15.106:0/652626919 shutdown_connections 2024-03-28T02:11:02.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.458+0000 7f1241509640 1 -- 172.21.15.106:0/652626919 wait complete. 2024-03-28T02:11:02.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-28T02:11:02.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.571+0000 7f7bb2f77640 1 -- 172.21.15.106:0/2814774380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac103280 msgr2=0x7f7bac109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.571+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/2814774380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac103280 0x7f7bac109b10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009be0 tx=0x7f7ba402bef0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.571+0000 7f7bb2f77640 1 -- 172.21.15.106:0/2814774380 shutdown_connections 2024-03-28T02:11:02.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.571+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/2814774380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac103280 0x7f7bac109b10 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.571+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/2814774380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bac1028c0 0x7f7bac102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.571+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/2814774380 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bac1016d0 0x7f7bac101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.571+0000 7f7bb2f77640 1 -- 172.21.15.106:0/2814774380 >> 172.21.15.106:0/2814774380 conn(0x7f7bac0fcc40 msgr2=0x7f7bac0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 -- 172.21.15.106:0/2814774380 shutdown_connections 2024-03-28T02:11:02.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 -- 172.21.15.106:0/2814774380 wait complete. 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 Processor -- start 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 -- start start 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bac1016d0 0x7f7bac197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1028c0 0x7f7bac197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bac103280 0x7f7bac19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac10d350 con 0x7f7bac1028c0 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac10d1d0 con 0x7f7bac1016d0 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb2f77640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac10d4d0 con 0x7f7bac103280 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7babfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1028c0 0x7f7bac197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb0d13640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bac1016d0 0x7f7bac197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7babfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1028c0 0x7f7bac197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54780/0 (socket says 172.21.15.106:54780) 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7babfff640 1 -- 172.21.15.106:0/1610981403 learned_addr learned my addr 172.21.15.106:0/1610981403 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.572+0000 7f7bb1514640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bac103280 0x7f7bac19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7babfff640 1 -- 172.21.15.106:0/1610981403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bac103280 msgr2=0x7f7bac19ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7babfff640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bac103280 0x7f7bac19ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7babfff640 1 -- 172.21.15.106:0/1610981403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bac1016d0 msgr2=0x7f7bac197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7babfff640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bac1016d0 0x7f7bac197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7babfff640 1 -- 172.21.15.106:0/1610981403 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac19f590 con 0x7f7bac1028c0 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bac1016d0 0x7f7bac197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7babfff640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1028c0 0x7f7bac197eb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800e9c0 tx=0x7f7b9800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7ba9ffb640 1 -- 172.21.15.106:0/1610981403 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9800ccc0 con 0x7f7bac1028c0 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7bb2f77640 1 -- 172.21.15.106:0/1610981403 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bac19f880 con 0x7f7bac1028c0 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7ba9ffb640 1 -- 172.21.15.106:0/1610981403 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b98005500 con 0x7f7bac1028c0 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7ba9ffb640 1 -- 172.21.15.106:0/1610981403 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98010430 con 0x7f7bac1028c0 2024-03-28T02:11:02.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.573+0000 7f7bb2f77640 1 -- 172.21.15.106:0/1610981403 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bac19fdd0 con 0x7f7bac1028c0 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7ba9ffb640 1 -- 172.21.15.106:0/1610981403 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7b980046c0 con 0x7f7bac1028c0 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb5b65740 1 Processor -- start 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb5b65740 1 -- start start 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e8b50 0x5627da9e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9ee040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 0x5627da9f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7ba9ffb640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7b9c0a2800 0x7f7b9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627da949c20 con 0x5627da9e9540 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627da8f2a40 con 0x5627da9e8b50 2024-03-28T02:11:02.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627da945cc0 con 0x5627da9ee6b0 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb0d13640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb1514640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7b9c0a2800 0x7f7b9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb0d13640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9ee040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54784/0 (socket says 172.21.15.106:54784) 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb0d13640 1 -- 172.21.15.106:0/3615433695 learned_addr learned my addr 172.21.15.106:0/3615433695 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7babfff640 1 --2- 172.21.15.106:0/3615433695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 0x5627da9f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.576+0000 7f7bb1514640 1 --2- 172.21.15.106:0/3615433695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e8b50 0x5627da9e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7ba9ffb640 1 -- 172.21.15.106:0/1610981403 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7b980cf0f0 con 0x7f7bac1028c0 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7bb0d13640 1 -- 172.21.15.106:0/3615433695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 msgr2=0x5627da9f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/3615433695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 0x5627da9f0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7bb0d13640 1 -- 172.21.15.106:0/3615433695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e8b50 msgr2=0x5627da9e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/3615433695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e8b50 0x5627da9e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7bb0d13640 1 -- 172.21.15.106:0/3615433695 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627da914f90 con 0x5627da9e9540 2024-03-28T02:11:02.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7bb1514640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7b9c0a2800 0x7f7b9c0a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4005f00 tx=0x7f7ba40098d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/3615433695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9ee040 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00114f0 tx=0x7f7ba00119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40efb253ed8b8fd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7b96ffd640 1 -- 172.21.15.106:0/3615433695 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0011250 con 0x5627da9e9540 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7b96ffd640 1 -- 172.21.15.106:0/3615433695 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ba00035f0 con 0x5627da9e9540 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.577+0000 7f7b96ffd640 1 -- 172.21.15.106:0/3615433695 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba000f800 con 0x5627da9e9540 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3615433695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 msgr2=0x5627da9ee040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3615433695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9ee040 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00114f0 tx=0x7f7ba00119b0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3615433695 shutdown_connections 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3615433695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 0x5627da9f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3615433695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9ee040 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3615433695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e8b50 0x5627da9e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3615433695 >> 172.21.15.106:0/3615433695 conn(0x5627da9aca60 msgr2=0x5627da9e00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3615433695 shutdown_connections 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3615433695 wait complete. 2024-03-28T02:11:02.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 -- start start 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 Processor -- start 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9f3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e9540 0x5627da9f3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 0x5627da9fa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627da9f1660 con 0x5627da9e8b50 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627da905e90 con 0x5627da9e9540 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.578+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627da9f17e0 con 0x5627da9ee6b0 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb0d13640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb5b65740 10 client.0 _ll_register_callbacks cb 0x5627da65dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb0d13640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9f3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54786/0 (socket says 172.21.15.106:54786) 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb0d13640 1 -- 172.21.15.106:0/3690318637 learned_addr learned my addr 172.21.15.106:0/3690318637 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb1514640 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 0x5627da9fa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7babfff640 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e9540 0x5627da9f3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb0d13640 1 -- 172.21.15.106:0/3690318637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 msgr2=0x5627da9fa5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 0x5627da9fa5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb0d13640 1 -- 172.21.15.106:0/3690318637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e9540 msgr2=0x5627da9f3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e9540 0x5627da9f3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb0d13640 1 -- 172.21.15.106:0/3690318637 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627da9129d0 con 0x5627da9e8b50 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9f3030 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7ba001b040 tx=0x7f7ba0011480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f590b05c062ba8c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7b8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7b8bfff640 1 -- 172.21.15.106:0/3690318637 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0028070 con 0x5627da9e8b50 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7b8bfff640 1 -- 172.21.15.106:0/3690318637 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7ba00041b0 con 0x5627da9e8b50 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3690318637 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ba00035f0 con 0x5627da9e8b50 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7b8bfff640 1 -- 172.21.15.106:0/3690318637 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ba0034050 con 0x5627da9e8b50 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb5b65740 10 client.15456 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.579+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3690318637 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5627da65e2d0 con 0x5627da9e8b50 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7b8bfff640 1 -- 172.21.15.106:0/3690318637 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ba00035f0 con 0x5627da9e8b50 2024-03-28T02:11:02.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7b8bfff640 1 -- 172.21.15.106:0/3690318637 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f7ba0004880 con 0x5627da9e8b50 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7bb5b65740 10 client.15456 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7bb5b65740 10 client.15456 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7bb5b65740 20 client.15456 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5627da9ff5a0 0x5627da9fe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7bb5b65740 4 client.15456 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3690318637 --> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5627da957160 con 0x5627da9ff5a0 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7babfff640 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5627da9ff5a0 0x5627da9fe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7b8bfff640 10 client.15456 ms_handle_connect on v2:172.21.15.153:6834/3469968496 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.580+0000 7f7babfff640 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5627da9ff5a0 0x5627da9fe7d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=738e520f3b2365c2 server_cookie=c5527933bddfbe47 in_seq=0 out_seq=0 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7b8bfff640 1 -- 172.21.15.106:0/3690318637 <== mds.3 v2:172.21.15.153:6834/3469968496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5627da957160 con 0x5627da9ff5a0 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7b8bfff640 10 client.15456 handle_command_reply: tid=0 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 client.15456 shutdown 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3690318637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 msgr2=0x5627da9f3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9f3030 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7ba001b040 tx=0x7f7ba0011480 comp rx=0 tx=0).stop 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3690318637 shutdown_connections 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5627da9ff5a0 0x5627da9fe7d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9ee6b0 0x5627da9fa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9e9540 0x5627da9f3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7b8bfff640 0 client.15456 ms_handle_reset on v2:172.21.15.153:6834/3469968496 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/3690318637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9f3030 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3690318637 >> 172.21.15.106:0/3690318637 conn(0x5627da9aca60 msgr2=0x5627da9df2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3690318637 shutdown_connections 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 1 -- 172.21.15.106:0/3690318637 wait complete. 2024-03-28T02:11:02.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.581+0000 7f7bb5b65740 20 client.15456 trim_cache size 0 max 16384 2024-03-28T02:11:02.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1719307200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:11:02.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3600882944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:11:02.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3121517077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:11:02.596 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb5b65740 1 Processor -- start 2024-03-28T02:11:02.596 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb5b65740 1 -- start start 2024-03-28T02:11:02.596 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9e3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.596 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e9540 0x5627da9fa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 0x5627daa07a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627da9fedc0 con 0x5627da9e8b50 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627da9ff420 con 0x5627da9ee6b0 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627da9ffa80 con 0x5627da9e9540 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb0d13640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9e3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb0d13640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9e3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54794/0 (socket says 172.21.15.106:54794) 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb0d13640 1 -- 172.21.15.106:0/909424140 learned_addr learned my addr 172.21.15.106:0/909424140 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7babfff640 1 --2- 172.21.15.106:0/909424140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e9540 0x5627da9fa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb1514640 1 --2- 172.21.15.106:0/909424140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 0x5627daa07a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb0d13640 1 -- 172.21.15.106:0/909424140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e9540 msgr2=0x5627da9fa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/909424140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e9540 0x5627da9fa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb0d13640 1 -- 172.21.15.106:0/909424140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 msgr2=0x5627daa07a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/909424140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 0x5627daa07a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.598+0000 7f7bb0d13640 1 -- 172.21.15.106:0/909424140 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627da7286b0 con 0x5627da9e8b50 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/909424140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9e3a00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0004ca0 tx=0x7f7ba0003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb86df21592e0571 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7b967fc640 1 -- 172.21.15.106:0/909424140 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba003d070 con 0x5627da9e8b50 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7b967fc640 1 -- 172.21.15.106:0/909424140 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ba0031030 con 0x5627da9e8b50 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7b967fc640 1 -- 172.21.15.106:0/909424140 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0028040 con 0x5627da9e8b50 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb5b65740 1 -- 172.21.15.106:0/909424140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 msgr2=0x5627da9e3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/909424140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9e3a00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0004ca0 tx=0x7f7ba0003ea0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb5b65740 1 -- 172.21.15.106:0/909424140 shutdown_connections 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/909424140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 0x5627daa07a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/909424140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e9540 0x5627da9fa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/909424140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e8b50 0x5627da9e3a00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb5b65740 1 -- 172.21.15.106:0/909424140 >> 172.21.15.106:0/909424140 conn(0x5627da9aca60 msgr2=0x5627da9e32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb5b65740 1 -- 172.21.15.106:0/909424140 shutdown_connections 2024-03-28T02:11:02.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.599+0000 7f7bb5b65740 1 -- 172.21.15.106:0/909424140 wait complete. 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb5b65740 1 -- start start 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb5b65740 1 Processor -- start 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e8b50 0x5627da9fa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb5b65740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 0x5627da9f7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627daa00740 con 0x5627da9e9540 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627daa000e0 con 0x5627da9ee6b0 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb5b65740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627daa00da0 con 0x5627da9e8b50 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb5b65740 10 client.0 _ll_register_callbacks cb 0x7f7ba0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7babfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7babfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9f6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54800/0 (socket says 172.21.15.106:54800) 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb1514640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 0x5627da9f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7babfff640 1 -- 172.21.15.106:0/4099186982 learned_addr learned my addr 172.21.15.106:0/4099186982 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e8b50 0x5627da9fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7babfff640 1 -- 172.21.15.106:0/4099186982 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e8b50 msgr2=0x5627da9fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7babfff640 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e8b50 0x5627da9fa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7babfff640 1 -- 172.21.15.106:0/4099186982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 msgr2=0x5627da9f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7babfff640 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 0x5627da9f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.600+0000 7f7babfff640 1 -- 172.21.15.106:0/4099186982 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627da9fa670 con 0x5627da9e9540 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7babfff640 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9f6b30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7b980cc850 tx=0x7f7b9800aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e5f2ae2daae91f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7b8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7b8bfff640 1 -- 172.21.15.106:0/4099186982 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9809bdf0 con 0x5627da9e9540 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7b8bfff640 1 -- 172.21.15.106:0/4099186982 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7b98086040 con 0x5627da9e9540 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb5b65740 1 -- 172.21.15.106:0/4099186982 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ba0031030 con 0x5627da9e9540 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7b8bfff640 1 -- 172.21.15.106:0/4099186982 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b9809b6f0 con 0x5627da9e9540 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb5b65740 10 client.15462 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:02.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb5b65740 1 -- 172.21.15.106:0/4099186982 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5627daa5b550 con 0x5627da9e9540 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7b8bfff640 1 -- 172.21.15.106:0/4099186982 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ba0031030 con 0x5627da9e9540 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7b8bfff640 1 -- 172.21.15.106:0/4099186982 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f7b980b5350 con 0x5627da9e9540 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb5b65740 10 client.15462 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb5b65740 10 client.15462 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb5b65740 20 client.15462 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5627daa5fc00 0x5627daa5b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb5b65740 4 client.15462 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb5b65740 1 -- 172.21.15.106:0/4099186982 --> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5627da9514b0 con 0x5627daa5fc00 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.601+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5627daa5fc00 0x5627daa5b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7b8bfff640 10 client.15462 ms_handle_connect on v2:172.21.15.153:6834/3469968496 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb0d13640 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5627daa5fc00 0x5627daa5b8e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=44450fce130e482c server_cookie=d8d73827cff6227a in_seq=0 out_seq=0 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7b8bfff640 1 -- 172.21.15.106:0/4099186982 <== mds.3 v2:172.21.15.153:6834/3469968496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5627da9514b0 con 0x5627daa5fc00 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7b8bfff640 10 client.15462 handle_command_reply: tid=0 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 client.15462 shutdown 2024-03-28T02:11:02.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 -- 172.21.15.106:0/4099186982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 msgr2=0x5627da9f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9f6b30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7b980cc850 tx=0x7f7b9800aea0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 -- 172.21.15.106:0/4099186982 shutdown_connections 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.153:6834/3469968496,v1:172.21.15.153:6835/3469968496] conn(0x5627daa5fc00 0x5627daa5b8e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5627da9ee6b0 0x5627da9f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5627da9e9540 0x5627da9f6b30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 --2- 172.21.15.106:0/4099186982 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5627da9e8b50 0x5627da9fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7b8bfff640 0 client.15462 ms_handle_reset on v2:172.21.15.153:6834/3469968496 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 -- 172.21.15.106:0/4099186982 >> 172.21.15.106:0/4099186982 conn(0x5627da9aca60 msgr2=0x5627da9dea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 -- 172.21.15.106:0/4099186982 shutdown_connections 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 1 -- 172.21.15.106:0/4099186982 wait complete. 2024-03-28T02:11:02.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.602+0000 7f7bb5b65740 20 client.15462 trim_cache size 0 max 16384 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 3, 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24446, 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 113.337274691, 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 93, 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 160.15477555000001 2024-03-28T02:11:02.603 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:11:02.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 -- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7b9c0a2800 msgr2=0x7f7b9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7b9c0a2800 0x7f7b9c0a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4005f00 tx=0x7f7ba40098d0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 -- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1028c0 msgr2=0x7f7bac197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1028c0 0x7f7bac197eb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800e9c0 tx=0x7f7b9800ee80 comp rx=0 tx=0).stop 2024-03-28T02:11:02.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 -- 172.21.15.106:0/1610981403 shutdown_connections 2024-03-28T02:11:02.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bac103280 0x7f7bac19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1028c0 0x7f7bac197eb0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f7b9c0a2800 0x7f7b9c0a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 --2- 172.21.15.106:0/1610981403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bac1016d0 0x7f7bac197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 -- 172.21.15.106:0/1610981403 >> 172.21.15.106:0/1610981403 conn(0x7f7bac0fcc40 msgr2=0x7f7bac105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 -- 172.21.15.106:0/1610981403 shutdown_connections 2024-03-28T02:11:02.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.603+0000 7f7bb2f77640 1 -- 172.21.15.106:0/1610981403 wait complete. 2024-03-28T02:11:02.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-28T02:11:02.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 -- 172.21.15.106:0/677995569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 msgr2=0x7f7104102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 --2- 172.21.15.106:0/677995569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 0x7f7104102d20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7100009900 tx=0x7f710002bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:02.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 -- 172.21.15.106:0/677995569 shutdown_connections 2024-03-28T02:11:02.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 --2- 172.21.15.106:0/677995569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7104103260 0x7f7104109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 --2- 172.21.15.106:0/677995569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 0x7f7104102d20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 --2- 172.21.15.106:0/677995569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71041016b0 0x7f7104101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 -- 172.21.15.106:0/677995569 >> 172.21.15.106:0/677995569 conn(0x7f71040fcc40 msgr2=0x7f71040ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 -- 172.21.15.106:0/677995569 shutdown_connections 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 -- 172.21.15.106:0/677995569 wait complete. 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.716+0000 7f710b412640 1 Processor -- start 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f710b412640 1 -- start start 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f710b412640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71041016b0 0x7f71041979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f710b412640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 0x7f7104197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f710b412640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7104103260 0x7f710419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f710b412640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710410d330 con 0x7f71041028a0 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f710b412640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710410d1b0 con 0x7f7104103260 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f710b412640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710410d4b0 con 0x7f71041016b0 2024-03-28T02:11:02.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71089ad640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 0x7f7104197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71089ad640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 0x7f7104197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54816/0 (socket says 172.21.15.106:54816) 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71089ad640 1 -- 172.21.15.106:0/626383478 learned_addr learned my addr 172.21.15.106:0/626383478 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71091ae640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71041016b0 0x7f71041979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71099af640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7104103260 0x7f710419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71089ad640 1 -- 172.21.15.106:0/626383478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71041016b0 msgr2=0x7f71041979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71089ad640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71041016b0 0x7f71041979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71089ad640 1 -- 172.21.15.106:0/626383478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7104103260 msgr2=0x7f710419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71089ad640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7104103260 0x7f710419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71089ad640 1 -- 172.21.15.106:0/626383478 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710419f600 con 0x7f71041028a0 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f71089ad640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 0x7f7104197f20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7100009230 tx=0x7f710002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f70fa7fc640 1 -- 172.21.15.106:0/626383478 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7100041070 con 0x7f71041028a0 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f70fa7fc640 1 -- 172.21.15.106:0/626383478 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7100004450 con 0x7f71041028a0 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f710b412640 1 -- 172.21.15.106:0/626383478 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710419f890 con 0x7f71041028a0 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f70fa7fc640 1 -- 172.21.15.106:0/626383478 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7100003db0 con 0x7f71041028a0 2024-03-28T02:11:02.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.717+0000 7f710b412640 1 -- 172.21.15.106:0/626383478 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710419fd80 con 0x7f71041028a0 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.720+0000 7f70fa7fc640 1 -- 172.21.15.106:0/626383478 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7100044070 con 0x7f71041028a0 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.720+0000 7f710dfff740 1 Processor -- start 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.720+0000 7f710dfff740 1 -- start start 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f710dfff740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 0x558d7a02db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f710dfff740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a032c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f710dfff740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 0x558d7a035730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f710dfff740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d79f92c20 con 0x558d7a02e130 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f710dfff740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d79f3ba40 con 0x558d7a02d740 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f710dfff740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d79f8ecc0 con 0x558d7a0332a0 2024-03-28T02:11:02.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f70fa7fc640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f70f00a2800 0x7f70f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71091ae640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a032c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71091ae640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a032c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54818/0 (socket says 172.21.15.106:54818) 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71091ae640 1 -- 172.21.15.106:0/2797771837 learned_addr learned my addr 172.21.15.106:0/2797771837 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71089ad640 1 --2- 172.21.15.106:0/2797771837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 0x558d7a035730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71099af640 1 --2- 172.21.15.106:0/2797771837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 0x558d7a02db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71099af640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f70f00a2800 0x7f70f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71091ae640 1 -- 172.21.15.106:0/2797771837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 msgr2=0x558d7a035730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71091ae640 1 --2- 172.21.15.106:0/2797771837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 0x558d7a035730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71091ae640 1 -- 172.21.15.106:0/2797771837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 msgr2=0x558d7a02db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71091ae640 1 --2- 172.21.15.106:0/2797771837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 0x558d7a02db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71091ae640 1 -- 172.21.15.106:0/2797771837 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d79f5df90 con 0x558d7a02e130 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f70fa7fc640 1 -- 172.21.15.106:0/626383478 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f71000f09b0 con 0x7f71041028a0 2024-03-28T02:11:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71099af640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f70f00a2800 0x7f70f00a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x558d7a02e990 tx=0x7f70fc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f71091ae640 1 --2- 172.21.15.106:0/2797771837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a032c30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009ca0 tx=0x7f70f4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a7d7747bc694ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.721+0000 7f70ef7fe640 1 -- 172.21.15.106:0/2797771837 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4004220 con 0x558d7a02e130 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f70ef7fe640 1 -- 172.21.15.106:0/2797771837 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70f40043c0 con 0x558d7a02e130 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f70ef7fe640 1 -- 172.21.15.106:0/2797771837 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f40198b0 con 0x558d7a02e130 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f710dfff740 1 -- 172.21.15.106:0/2797771837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 msgr2=0x558d7a032c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f710dfff740 1 --2- 172.21.15.106:0/2797771837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a032c30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009ca0 tx=0x7f70f4011810 comp rx=0 tx=0).stop 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f710dfff740 1 -- 172.21.15.106:0/2797771837 shutdown_connections 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f710dfff740 1 --2- 172.21.15.106:0/2797771837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 0x558d7a035730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f710dfff740 1 --2- 172.21.15.106:0/2797771837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a032c30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f710dfff740 1 --2- 172.21.15.106:0/2797771837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 0x558d7a02db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f710dfff740 1 -- 172.21.15.106:0/2797771837 >> 172.21.15.106:0/2797771837 conn(0x558d79ff5a60 msgr2=0x558d7a0290b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f710dfff740 1 -- 172.21.15.106:0/2797771837 shutdown_connections 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.722+0000 7f710dfff740 1 -- 172.21.15.106:0/2797771837 wait complete. 2024-03-28T02:11:02.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f710dfff740 1 -- start start 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f710dfff740 1 Processor -- start 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f710dfff740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02d740 0x558d7a037c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f710dfff740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a038160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f710dfff740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 0x558d7a03f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f710dfff740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d7a036250 con 0x558d7a02e130 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f710dfff740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d79f4ee90 con 0x558d7a0332a0 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f710dfff740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d7a0363d0 con 0x558d7a02d740 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71089ad640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a038160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f710dfff740 10 client.0 _ll_register_callbacks cb 0x558d79ca6ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71089ad640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a038160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54832/0 (socket says 172.21.15.106:54832) 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71091ae640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02d740 0x558d7a037c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71089ad640 1 -- 172.21.15.106:0/2223370844 learned_addr learned my addr 172.21.15.106:0/2223370844 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71099af640 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 0x558d7a03f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71089ad640 1 -- 172.21.15.106:0/2223370844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02d740 msgr2=0x558d7a037c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71089ad640 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02d740 0x558d7a037c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71089ad640 1 -- 172.21.15.106:0/2223370844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 msgr2=0x558d7a03f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71089ad640 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 0x558d7a03f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.723+0000 7f71089ad640 1 -- 172.21.15.106:0/2223370844 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d79f5b9d0 con 0x558d7a02e130 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f71089ad640 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a038160 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7100006010 tx=0x7f71000058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541a51a1c764e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f70ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f70ecff9640 1 -- 172.21.15.106:0/2223370844 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000f0e60 con 0x558d7a02e130 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f70ecff9640 1 -- 172.21.15.106:0/2223370844 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f71000035f0 con 0x558d7a02e130 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f710dfff740 1 -- 172.21.15.106:0/2223370844 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f70f40043c0 con 0x558d7a02e130 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f70ecff9640 1 -- 172.21.15.106:0/2223370844 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f710004b040 con 0x558d7a02e130 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f710dfff740 10 client.15474 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:02.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f710dfff740 1 -- 172.21.15.106:0/2223370844 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558d79ca72d0 con 0x558d7a02e130 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f70ecff9640 1 -- 172.21.15.106:0/2223370844 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70f40043c0 con 0x558d7a02e130 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f70ecff9640 1 -- 172.21.15.106:0/2223370844 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f71000b9620 con 0x558d7a02e130 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f710dfff740 10 client.15474 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f710dfff740 10 client.15474 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f710dfff740 20 client.15474 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f710dfff740 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x558d7a0443e0 0x558d7a043300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f710dfff740 4 client.15474 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f710dfff740 1 -- 172.21.15.106:0/2223370844 --> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558d79fa0160 con 0x558d7a0443e0 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.724+0000 7f71091ae640 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x558d7a0443e0 0x558d7a043300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.725+0000 7f71091ae640 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x558d7a0443e0 0x558d7a043300 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f76fc5fbb8a2feca server_cookie=c7036d794805ec9a in_seq=0 out_seq=0 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.725+0000 7f70ecff9640 10 client.15474 ms_handle_connect on v2:172.21.15.178:6834/1402366472 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.725+0000 7f70ecff9640 1 -- 172.21.15.106:0/2223370844 <== mds.2 v2:172.21.15.178:6834/1402366472 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558d79fa0160 con 0x558d7a0443e0 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.725+0000 7f70ecff9640 10 client.15474 handle_command_reply: tid=0 2024-03-28T02:11:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.725+0000 7f710dfff740 1 client.15474 shutdown 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.725+0000 7f710dfff740 1 -- 172.21.15.106:0/2223370844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 msgr2=0x558d7a038160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.725+0000 7f710dfff740 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a038160 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7100006010 tx=0x7f71000058d0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.725+0000 7f710dfff740 1 -- 172.21.15.106:0/2223370844 shutdown_connections 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.726+0000 7f710dfff740 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x558d7a0443e0 0x558d7a043300 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.726+0000 7f710dfff740 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 0x558d7a03f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.726+0000 7f710dfff740 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a038160 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.726+0000 7f710dfff740 1 --2- 172.21.15.106:0/2223370844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02d740 0x558d7a037c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.726+0000 7f70ecff9640 0 client.15474 ms_handle_reset on v2:172.21.15.178:6834/1402366472 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.726+0000 7f710dfff740 1 -- 172.21.15.106:0/2223370844 >> 172.21.15.106:0/2223370844 conn(0x558d79ff5a60 msgr2=0x558d7a0282f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.726+0000 7f710dfff740 1 -- 172.21.15.106:0/2223370844 shutdown_connections 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.726+0000 7f710dfff740 1 -- 172.21.15.106:0/2223370844 wait complete. 2024-03-28T02:11:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.726+0000 7f710dfff740 20 client.15474 trim_cache size 0 max 16384 2024-03-28T02:11:02.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1719307200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:11:02.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3600882944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:11:02.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3121517077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f710dfff740 1 Processor -- start 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f710dfff740 1 -- start start 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f710dfff740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 0x558d7a036ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f710dfff740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a03f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f710dfff740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 0x558d7a037430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f710dfff740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d7a043c00 con 0x558d7a02e130 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f710dfff740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d7a044260 con 0x558d7a02d740 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f710dfff740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d7a0448c0 con 0x558d7a0332a0 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f71089ad640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a03f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f71089ad640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a03f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54846/0 (socket says 172.21.15.106:54846) 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f71089ad640 1 -- 172.21.15.106:0/3067577997 learned_addr learned my addr 172.21.15.106:0/3067577997 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f71091ae640 1 --2- 172.21.15.106:0/3067577997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 0x558d7a036ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.742+0000 7f71099af640 1 --2- 172.21.15.106:0/3067577997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 0x558d7a037430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f71089ad640 1 -- 172.21.15.106:0/3067577997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 msgr2=0x558d7a037430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f71089ad640 1 --2- 172.21.15.106:0/3067577997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 0x558d7a037430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f71089ad640 1 -- 172.21.15.106:0/3067577997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 msgr2=0x558d7a036ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f71089ad640 1 --2- 172.21.15.106:0/3067577997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 0x558d7a036ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f71089ad640 1 -- 172.21.15.106:0/3067577997 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d79d716b0 con 0x558d7a02e130 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f71089ad640 1 --2- 172.21.15.106:0/3067577997 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a03f160 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7100009430 tx=0x7f71000bb4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e135c18c1b984f9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f70eeffd640 1 -- 172.21.15.106:0/3067577997 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000c52c0 con 0x558d7a02e130 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f70eeffd640 1 -- 172.21.15.106:0/3067577997 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71000d1e50 con 0x558d7a02e130 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f70eeffd640 1 -- 172.21.15.106:0/3067577997 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000bc910 con 0x558d7a02e130 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f710dfff740 1 -- 172.21.15.106:0/3067577997 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 msgr2=0x558d7a03f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f710dfff740 1 --2- 172.21.15.106:0/3067577997 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a03f160 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7100009430 tx=0x7f71000bb4c0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f710dfff740 1 -- 172.21.15.106:0/3067577997 shutdown_connections 2024-03-28T02:11:02.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f710dfff740 1 --2- 172.21.15.106:0/3067577997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a0332a0 0x558d7a037430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f710dfff740 1 --2- 172.21.15.106:0/3067577997 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02e130 0x558d7a03f160 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f710dfff740 1 --2- 172.21.15.106:0/3067577997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a02d740 0x558d7a036ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f710dfff740 1 -- 172.21.15.106:0/3067577997 >> 172.21.15.106:0/3067577997 conn(0x558d79ff5a60 msgr2=0x558d7a037c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f710dfff740 1 -- 172.21.15.106:0/3067577997 shutdown_connections 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.743+0000 7f710dfff740 1 -- 172.21.15.106:0/3067577997 wait complete. 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f710dfff740 1 -- start start 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f710dfff740 1 Processor -- start 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f710dfff740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02d740 0x558d7a03f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f710dfff740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02e130 0x558d7a03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f710dfff740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 0x558d7a03c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f710dfff740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d7a045580 con 0x558d7a02d740 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f710dfff740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d7a044f20 con 0x558d7a0332a0 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f710dfff740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d7a045be0 con 0x558d7a02e130 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f710dfff740 10 client.0 _ll_register_callbacks cb 0x7f7100003f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:02.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71091ae640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02d740 0x558d7a03f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71091ae640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02d740 0x558d7a03f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54862/0 (socket says 172.21.15.106:54862) 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71091ae640 1 -- 172.21.15.106:0/1009147948 learned_addr learned my addr 172.21.15.106:0/1009147948 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71099af640 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 0x558d7a03c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71089ad640 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02e130 0x558d7a03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71091ae640 1 -- 172.21.15.106:0/1009147948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02e130 msgr2=0x558d7a03bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71091ae640 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02e130 0x558d7a03bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71091ae640 1 -- 172.21.15.106:0/1009147948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 msgr2=0x558d7a03c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71091ae640 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 0x558d7a03c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71091ae640 1 -- 172.21.15.106:0/1009147948 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d7a09f440 con 0x558d7a02d740 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.744+0000 7f71089ad640 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02e130 0x558d7a03bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f71091ae640 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02d740 0x558d7a03f120 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009bc0 tx=0x7f70f4005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b473ecdc7bba1387 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f70ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f70ecff9640 1 -- 172.21.15.106:0/1009147948 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4005290 con 0x558d7a02d740 2024-03-28T02:11:02.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f70ecff9640 1 -- 172.21.15.106:0/1009147948 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f70f4005430 con 0x558d7a02d740 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f710dfff740 1 -- 172.21.15.106:0/1009147948 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f71000d1e50 con 0x558d7a02d740 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f70ecff9640 1 -- 172.21.15.106:0/1009147948 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70f403b050 con 0x558d7a02d740 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f710dfff740 10 client.15480 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f710dfff740 1 -- 172.21.15.106:0/1009147948 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558d7a0a06e0 con 0x558d7a02d740 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f70ecff9640 1 -- 172.21.15.106:0/1009147948 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71000d1e50 con 0x558d7a02d740 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f70ecff9640 1 -- 172.21.15.106:0/1009147948 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f70f4019790 con 0x558d7a02d740 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f710dfff740 10 client.15480 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f710dfff740 10 client.15480 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f710dfff740 20 client.15480 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f710dfff740 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x558d7a0a4bb0 0x558d7a0a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f710dfff740 4 client.15480 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f710dfff740 1 -- 172.21.15.106:0/1009147948 --> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558d79f9a4b0 con 0x558d7a0a4bb0 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.745+0000 7f71089ad640 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x558d7a0a4bb0 0x558d7a0a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f70ecff9640 10 client.15480 ms_handle_connect on v2:172.21.15.178:6834/1402366472 2024-03-28T02:11:02.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f71089ad640 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x558d7a0a4bb0 0x558d7a0a09d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f580d6a565188a4d server_cookie=9aea9d8d56735dda in_seq=0 out_seq=0 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f70ecff9640 1 -- 172.21.15.106:0/1009147948 <== mds.2 v2:172.21.15.178:6834/1402366472 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x558d79f9a4b0 con 0x558d7a0a4bb0 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f70ecff9640 10 client.15480 handle_command_reply: tid=0 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 client.15480 shutdown 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 -- 172.21.15.106:0/1009147948 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02d740 msgr2=0x558d7a03f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02d740 0x558d7a03f120 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009bc0 tx=0x7f70f4005d30 comp rx=0 tx=0).stop 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 -- 172.21.15.106:0/1009147948 shutdown_connections 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.178:6834/1402366472,v1:172.21.15.178:6835/1402366472] conn(0x558d7a0a4bb0 0x558d7a0a09d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d7a0332a0 0x558d7a03c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558d7a02e130 0x558d7a03bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f70ecff9640 0 client.15480 ms_handle_reset on v2:172.21.15.178:6834/1402366472 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 --2- 172.21.15.106:0/1009147948 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d7a02d740 0x558d7a03f120 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 -- 172.21.15.106:0/1009147948 >> 172.21.15.106:0/1009147948 conn(0x558d79ff5a60 msgr2=0x558d7a036ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 -- 172.21.15.106:0/1009147948 shutdown_connections 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 1 -- 172.21.15.106:0/1009147948 wait complete. 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.746+0000 7f710dfff740 20 client.15480 trim_cache size 0 max 16384 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 2, 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24460, 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 114.498088799, 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:11:02.747 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 93, 2024-03-28T02:11:02.748 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:11:02.748 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 157.228061639 2024-03-28T02:11:02.748 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:11:02.748 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 -- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f70f00a2800 msgr2=0x7f70f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f70f00a2800 0x7f70f00a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x558d7a02e990 tx=0x7f70fc009500 comp rx=0 tx=0).stop 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 -- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 msgr2=0x7f7104197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 0x7f7104197f20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7100009230 tx=0x7f710002c650 comp rx=0 tx=0).stop 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 -- 172.21.15.106:0/626383478 shutdown_connections 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f70f00a2800 0x7f70f00a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7104103260 0x7f710419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71041028a0 0x7f7104197f20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 --2- 172.21.15.106:0/626383478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71041016b0 0x7f71041979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 -- 172.21.15.106:0/626383478 >> 172.21.15.106:0/626383478 conn(0x7f71040fcc40 msgr2=0x7f7104105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 -- 172.21.15.106:0/626383478 shutdown_connections 2024-03-28T02:11:02.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.747+0000 7f710b412640 1 -- 172.21.15.106:0/626383478 wait complete. 2024-03-28T02:11:02.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T02:11:02.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 -- 172.21.15.106:0/397395039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54301028a0 msgr2=0x7f5430102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 --2- 172.21.15.106:0/397395039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54301028a0 0x7f5430102d20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f541c009900 tx=0x7f541c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:02.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 -- 172.21.15.106:0/397395039 shutdown_connections 2024-03-28T02:11:02.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 --2- 172.21.15.106:0/397395039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5430103260 0x7f5430109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 --2- 172.21.15.106:0/397395039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54301028a0 0x7f5430102d20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 --2- 172.21.15.106:0/397395039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54301016b0 0x7f5430101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 -- 172.21.15.106:0/397395039 >> 172.21.15.106:0/397395039 conn(0x7f54300fcc40 msgr2=0x7f54300ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 -- 172.21.15.106:0/397395039 shutdown_connections 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 -- 172.21.15.106:0/397395039 wait complete. 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 Processor -- start 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 -- start start 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54301016b0 0x7f5430197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54301028a0 0x7f5430197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5430103260 0x7f543019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543010d310 con 0x7f5430103260 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543010d190 con 0x7f54301016b0 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.861+0000 7f5436edc640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543010d490 con 0x7f54301028a0 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5435479640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5430103260 0x7f543019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5435479640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5430103260 0x7f543019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54874/0 (socket says 172.21.15.106:54874) 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5435479640 1 -- 172.21.15.106:0/1512421591 learned_addr learned my addr 172.21.15.106:0/1512421591 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f542ffff640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54301028a0 0x7f5430197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5434c78640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54301016b0 0x7f5430197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5435479640 1 -- 172.21.15.106:0/1512421591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54301028a0 msgr2=0x7f5430197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5435479640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54301028a0 0x7f5430197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5435479640 1 -- 172.21.15.106:0/1512421591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54301016b0 msgr2=0x7f5430197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5435479640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54301016b0 0x7f5430197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5435479640 1 -- 172.21.15.106:0/1512421591 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543019f630 con 0x7f5430103260 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5435479640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5430103260 0x7f543019efc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f542800efd0 tx=0x7f542800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f542dffb640 1 -- 172.21.15.106:0/1512421591 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428003b70 con 0x7f5430103260 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f542dffb640 1 -- 172.21.15.106:0/1512421591 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5428003d10 con 0x7f5430103260 2024-03-28T02:11:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5436edc640 1 -- 172.21.15.106:0/1512421591 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543019f920 con 0x7f5430103260 2024-03-28T02:11:02.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f542dffb640 1 -- 172.21.15.106:0/1512421591 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54280109f0 con 0x7f5430103260 2024-03-28T02:11:02.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.862+0000 7f5436edc640 1 -- 172.21.15.106:0/1512421591 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543019fdf0 con 0x7f5430103260 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.864+0000 7f542dffb640 1 -- 172.21.15.106:0/1512421591 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5428004020 con 0x7f5430103260 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5439b9f740 1 Processor -- start 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f542dffb640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f54200a2800 0x7f54200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5439b9f740 1 -- start start 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e447740 0x55c29e447b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e44d2a0 0x55c29e44f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f542ffff640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f54200a2800 0x7f54200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5439b9f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e3acc20 con 0x55c29e448130 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5439b9f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e355a40 con 0x55c29e447740 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5439b9f740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e3a8cc0 con 0x55c29e44d2a0 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f542dffb640 1 -- 172.21.15.106:0/1512421591 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f54280cf270 con 0x7f5430103260 2024-03-28T02:11:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f542ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5434c78640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e447740 0x55c29e447b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5435479640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e44d2a0 0x55c29e44f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f542ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54876/0 (socket says 172.21.15.106:54876) 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5434c78640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e447740 0x55c29e447b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:58548/0 (socket says 172.21.15.106:58548) 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f542ffff640 1 -- 172.21.15.106:0/1203464891 learned_addr learned my addr 172.21.15.106:0/1203464891 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f542ffff640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f54200a2800 0x7f54200a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f541c0096c0 tx=0x7f541c02cb90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5434c78640 1 -- 172.21.15.106:0/1203464891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e44d2a0 msgr2=0x55c29e44f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5434c78640 1 --2- 172.21.15.106:0/1203464891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e44d2a0 0x55c29e44f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5434c78640 1 -- 172.21.15.106:0/1203464891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 msgr2=0x55c29e44cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5434c78640 1 --2- 172.21.15.106:0/1203464891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5434c78640 1 -- 172.21.15.106:0/1203464891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c29e377f90 con 0x55c29e447740 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f5435479640 1 --2- 172.21.15.106:0/1203464891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e44d2a0 0x55c29e44f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.865+0000 7f542ffff640 1 --2- 172.21.15.106:0/1203464891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:02.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5434c78640 1 --2- 172.21.15.106:0/1203464891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e447740 0x55c29e447b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5424009bf0 tx=0x7f5424011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2f3456df6dcc4ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f541affd640 1 -- 172.21.15.106:0/1203464891 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424004050 con 0x55c29e447740 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f541affd640 1 -- 172.21.15.106:0/1203464891 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5424005500 con 0x55c29e447740 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f541affd640 1 -- 172.21.15.106:0/1203464891 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424004d10 con 0x55c29e447740 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5439b9f740 1 -- 172.21.15.106:0/1203464891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e447740 msgr2=0x55c29e447b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5439b9f740 1 --2- 172.21.15.106:0/1203464891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e447740 0x55c29e447b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5424009bf0 tx=0x7f5424011810 comp rx=0 tx=0).stop 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5439b9f740 1 -- 172.21.15.106:0/1203464891 shutdown_connections 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5439b9f740 1 --2- 172.21.15.106:0/1203464891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e44d2a0 0x55c29e44f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5439b9f740 1 --2- 172.21.15.106:0/1203464891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5439b9f740 1 --2- 172.21.15.106:0/1203464891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e447740 0x55c29e447b60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5439b9f740 1 -- 172.21.15.106:0/1203464891 >> 172.21.15.106:0/1203464891 conn(0x55c29e40fa60 msgr2=0x55c29e4430b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5439b9f740 1 -- 172.21.15.106:0/1203464891 shutdown_connections 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.866+0000 7f5439b9f740 1 -- 172.21.15.106:0/1203464891 wait complete. 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5439b9f740 1 -- start start 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5439b9f740 1 Processor -- start 2024-03-28T02:11:02.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e451c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e452160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5439b9f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e450250 con 0x55c29e44d2a0 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5439b9f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e368e90 con 0x55c29e448130 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5439b9f740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e4503d0 con 0x55c29e447740 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5439b9f740 10 client.0 _ll_register_callbacks cb 0x55c29e0c0ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5435479640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f542ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e452160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5435479640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54882/0 (socket says 172.21.15.106:54882) 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5435479640 1 -- 172.21.15.106:0/2059136651 learned_addr learned my addr 172.21.15.106:0/2059136651 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.867+0000 7f5434c78640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e451c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5435479640 1 -- 172.21.15.106:0/2059136651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 msgr2=0x55c29e451c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5435479640 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e451c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5435479640 1 -- 172.21.15.106:0/2059136651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 msgr2=0x55c29e452160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5435479640 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e452160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5435479640 1 -- 172.21.15.106:0/2059136651 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c29e3759d0 con 0x55c29e44d2a0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f542ffff640 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e452160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5434c78640 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e451c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5435479640 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4591a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f54280ccd40 tx=0x7f5428009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f5282ee46e613 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f540ffff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f540ffff640 1 -- 172.21.15.106:0/2059136651 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54280b4020 con 0x55c29e44d2a0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f540ffff640 1 -- 172.21.15.106:0/2059136651 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5428004620 con 0x55c29e44d2a0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5439b9f740 1 -- 172.21.15.106:0/2059136651 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5424005500 con 0x55c29e44d2a0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f540ffff640 1 -- 172.21.15.106:0/2059136651 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f542809f9c0 con 0x55c29e44d2a0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5439b9f740 10 client.15492 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5439b9f740 1 -- 172.21.15.106:0/2059136651 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c29e0c12d0 con 0x55c29e44d2a0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f540ffff640 1 -- 172.21.15.106:0/2059136651 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5424005500 con 0x55c29e44d2a0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f540ffff640 1 -- 172.21.15.106:0/2059136651 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f542801f020 con 0x55c29e44d2a0 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5439b9f740 10 client.15492 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:02.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5439b9f740 10 client.15492 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5439b9f740 20 client.15492 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x55c29e45e470 0x55c29e45d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5439b9f740 4 client.15492 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.868+0000 7f5439b9f740 1 -- 172.21.15.106:0/2059136651 --> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c29e3ba160 con 0x55c29e45e470 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.869+0000 7f5434c78640 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x55c29e45e470 0x55c29e45d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.869+0000 7f5434c78640 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x55c29e45e470 0x55c29e45d4e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5a14b2fda1951a6c server_cookie=e4542a5191ffac67 in_seq=0 out_seq=0 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.869+0000 7f540ffff640 10 client.15492 ms_handle_connect on v2:172.21.15.178:6838/937216083 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.869+0000 7f540ffff640 1 -- 172.21.15.106:0/2059136651 <== mds.1 v2:172.21.15.178:6838/937216083 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c29e3ba160 con 0x55c29e45e470 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.869+0000 7f540ffff640 10 client.15492 handle_command_reply: tid=0 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.869+0000 7f5439b9f740 1 client.15492 shutdown 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 -- 172.21.15.106:0/2059136651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 msgr2=0x55c29e4591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4591a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f54280ccd40 tx=0x7f5428009960 comp rx=0 tx=0).stop 2024-03-28T02:11:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 -- 172.21.15.106:0/2059136651 shutdown_connections 2024-03-28T02:11:02.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x55c29e45e470 0x55c29e45d4e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4591a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e452160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2059136651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e451c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 -- 172.21.15.106:0/2059136651 >> 172.21.15.106:0/2059136651 conn(0x55c29e40fa60 msgr2=0x55c29e4422f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f540ffff640 0 client.15492 ms_handle_reset on v2:172.21.15.178:6838/937216083 2024-03-28T02:11:02.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 -- 172.21.15.106:0/2059136651 shutdown_connections 2024-03-28T02:11:02.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 1 -- 172.21.15.106:0/2059136651 wait complete. 2024-03-28T02:11:02.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.870+0000 7f5439b9f740 20 client.15492 trim_cache size 0 max 16384 2024-03-28T02:11:02.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1719307200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:11:02.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3600882944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:11:02.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3121517077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5439b9f740 1 Processor -- start 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5439b9f740 1 -- start start 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e451240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e44d2a0 0x55c29e450800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5439b9f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e45dc90 con 0x55c29e448130 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5439b9f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e45e2f0 con 0x55c29e44d2a0 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5439b9f740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e45e950 con 0x55c29e447740 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f542ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f542ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54896/0 (socket says 172.21.15.106:54896) 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f542ffff640 1 -- 172.21.15.106:0/2338233363 learned_addr learned my addr 172.21.15.106:0/2338233363 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5434c78640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e451240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.886+0000 7f5435479640 1 --2- 172.21.15.106:0/2338233363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e44d2a0 0x55c29e450800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f542ffff640 1 -- 172.21.15.106:0/2338233363 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 msgr2=0x55c29e451240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f542ffff640 1 --2- 172.21.15.106:0/2338233363 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e451240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f542ffff640 1 -- 172.21.15.106:0/2338233363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e44d2a0 msgr2=0x55c29e450800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f542ffff640 1 --2- 172.21.15.106:0/2338233363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e44d2a0 0x55c29e450800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f542ffff640 1 -- 172.21.15.106:0/2338233363 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c29e18b6b0 con 0x55c29e448130 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f542ffff640 1 --2- 172.21.15.106:0/2338233363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44f690 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f541c02cfd0 tx=0x7f541c045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac708f62f1ebd9b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f541a7fc640 1 -- 172.21.15.106:0/2338233363 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c005240 con 0x55c29e448130 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f541a7fc640 1 -- 172.21.15.106:0/2338233363 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f541c0044d0 con 0x55c29e448130 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f541a7fc640 1 -- 172.21.15.106:0/2338233363 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c003850 con 0x55c29e448130 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f5439b9f740 1 -- 172.21.15.106:0/2338233363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 msgr2=0x55c29e44f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2338233363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44f690 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f541c02cfd0 tx=0x7f541c045000 comp rx=0 tx=0).stop 2024-03-28T02:11:02.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f5439b9f740 1 -- 172.21.15.106:0/2338233363 shutdown_connections 2024-03-28T02:11:02.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.887+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2338233363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e44d2a0 0x55c29e450800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2338233363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e448130 0x55c29e44f690 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2338233363 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e451240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 -- 172.21.15.106:0/2338233363 >> 172.21.15.106:0/2338233363 conn(0x55c29e40fa60 msgr2=0x55c29e45d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 -- 172.21.15.106:0/2338233363 shutdown_connections 2024-03-28T02:11:02.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 -- 172.21.15.106:0/2338233363 wait complete. 2024-03-28T02:11:02.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 -- start start 2024-03-28T02:11:02.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 Processor -- start 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e4b74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e4b9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4bbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e45f610 con 0x55c29e44d2a0 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e45efb0 con 0x55c29e448130 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e45fc70 con 0x55c29e447740 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5439b9f740 10 client.0 _ll_register_callbacks cb 0x7f5428004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5435479640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4bbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5434c78640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e4b74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5434c78640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e4b74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:39494/0 (socket says 172.21.15.106:39494) 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f5434c78640 1 -- 172.21.15.106:0/2850900141 learned_addr learned my addr 172.21.15.106:0/2850900141 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.888+0000 7f542ffff640 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e4b9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5435479640 1 -- 172.21.15.106:0/2850900141 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 msgr2=0x55c29e4b74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5435479640 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e4b74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5435479640 1 -- 172.21.15.106:0/2850900141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 msgr2=0x55c29e4b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5435479640 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e4b9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5435479640 1 -- 172.21.15.106:0/2850900141 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c29e4bc600 con 0x55c29e44d2a0 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f542ffff640 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e4b9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5435479640 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4bbf60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f54280981f0 tx=0x7f5428009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d09f55c3d77b5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f540ffff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f540ffff640 1 -- 172.21.15.106:0/2850900141 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54280b4040 con 0x55c29e44d2a0 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f540ffff640 1 -- 172.21.15.106:0/2850900141 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5428004b40 con 0x55c29e44d2a0 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5439b9f740 1 -- 172.21.15.106:0/2850900141 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f541c0044d0 con 0x55c29e44d2a0 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f540ffff640 1 -- 172.21.15.106:0/2850900141 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f542809b1d0 con 0x55c29e44d2a0 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5439b9f740 10 client.15498 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:02.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5439b9f740 1 -- 172.21.15.106:0/2850900141 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c29e4bd950 con 0x55c29e44d2a0 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f540ffff640 1 -- 172.21.15.106:0/2850900141 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f541c0044d0 con 0x55c29e44d2a0 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f540ffff640 1 -- 172.21.15.106:0/2850900141 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f5428083050 con 0x55c29e44d2a0 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5439b9f740 10 client.15498 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5439b9f740 10 client.15498 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5439b9f740 20 client.15498 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x55c29e4c1fa0 0x55c29e4bdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5439b9f740 4 client.15498 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.889+0000 7f5439b9f740 1 -- 172.21.15.106:0/2850900141 --> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c29e3b44b0 con 0x55c29e4c1fa0 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5434c78640 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x55c29e4c1fa0 0x55c29e4bdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5434c78640 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x55c29e4c1fa0 0x55c29e4bdbe0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=89c2884f5dfe6e55 server_cookie=3ef0d77aefe2fe6a in_seq=0 out_seq=0 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f540ffff640 10 client.15498 ms_handle_connect on v2:172.21.15.178:6838/937216083 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f540ffff640 1 -- 172.21.15.106:0/2850900141 <== mds.1 v2:172.21.15.178:6838/937216083 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c29e3b44b0 con 0x55c29e4c1fa0 2024-03-28T02:11:02.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f540ffff640 10 client.15498 handle_command_reply: tid=0 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 client.15498 shutdown 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 -- 172.21.15.106:0/2850900141 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 msgr2=0x55c29e4bbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4bbf60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f54280981f0 tx=0x7f5428009570 comp rx=0 tx=0).stop 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 -- 172.21.15.106:0/2850900141 shutdown_connections 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.178:6838/937216083,v1:172.21.15.178:6839/937216083] conn(0x55c29e4c1fa0 0x55c29e4bdbe0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c29e44d2a0 0x55c29e4bbf60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f540ffff640 0 client.15498 ms_handle_reset on v2:172.21.15.178:6838/937216083 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c29e448130 0x55c29e4b9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 --2- 172.21.15.106:0/2850900141 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55c29e447740 0x55c29e4b74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 -- 172.21.15.106:0/2850900141 >> 172.21.15.106:0/2850900141 conn(0x55c29e40fa60 msgr2=0x55c29e449590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 -- 172.21.15.106:0/2850900141 shutdown_connections 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.890+0000 7f5439b9f740 1 -- 172.21.15.106:0/2850900141 wait complete. 2024-03-28T02:11:02.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5439b9f740 20 client.15498 trim_cache size 0 max 16384 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 1, 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24499, 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 115.367941829, 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 93, 2024-03-28T02:11:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:11:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 148.84745434999999 2024-03-28T02:11:02.892 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:11:02.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 -- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f54200a2800 msgr2=0x7f54200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f54200a2800 0x7f54200a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f541c0096c0 tx=0x7f541c02cb90 comp rx=0 tx=0).stop 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 -- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5430103260 msgr2=0x7f543019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5430103260 0x7f543019efc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f542800efd0 tx=0x7f542800c6a0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 -- 172.21.15.106:0/1512421591 shutdown_connections 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5430103260 0x7f543019efc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54301028a0 0x7f5430197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f54200a2800 0x7f54200a4cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 --2- 172.21.15.106:0/1512421591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54301016b0 0x7f5430197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 -- 172.21.15.106:0/1512421591 >> 172.21.15.106:0/1512421591 conn(0x7f54300fcc40 msgr2=0x7f5430105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 -- 172.21.15.106:0/1512421591 shutdown_connections 2024-03-28T02:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:02.891+0000 7f5436edc640 1 -- 172.21.15.106:0/1512421591 wait complete. 2024-03-28T02:11:02.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T02:11:02.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 -- 172.21.15.106:0/301717580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 msgr2=0x7fbd70100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:02.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 --2- 172.21.15.106:0/301717580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 0x7fbd70100ca0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009900 tx=0x7fbd6002bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:02.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 -- 172.21.15.106:0/301717580 shutdown_connections 2024-03-28T02:11:02.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 --2- 172.21.15.106:0/301717580 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd701011e0 0x7fbd70108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 --2- 172.21.15.106:0/301717580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 0x7fbd70100ca0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 --2- 172.21.15.106:0/301717580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd70101bd0 0x7fbd70101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:02.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 -- 172.21.15.106:0/301717580 >> 172.21.15.106:0/301717580 conn(0x7fbd700fb2b0 msgr2=0x7fbd700fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:02.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 -- 172.21.15.106:0/301717580 shutdown_connections 2024-03-28T02:11:02.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 -- 172.21.15.106:0/301717580 wait complete. 2024-03-28T02:11:03.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 Processor -- start 2024-03-28T02:11:03.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.005+0000 7fbd74859640 1 -- start start 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd74859640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 0x7fbd70197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd74859640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd701011e0 0x7fbd70197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd74859640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd70101bd0 0x7fbd7019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd74859640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7010d3e0 con 0x7fbd70100820 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd74859640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7010d260 con 0x7fbd70101bd0 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd74859640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7010d560 con 0x7fbd701011e0 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6e59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 0x7fbd70197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6e59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 0x7fbd70197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54930/0 (socket says 172.21.15.106:54930) 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6e59c640 1 -- 172.21.15.106:0/1008842943 learned_addr learned my addr 172.21.15.106:0/1008842943 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd701011e0 0x7fbd70197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd70101bd0 0x7fbd7019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6e59c640 1 -- 172.21.15.106:0/1008842943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd701011e0 msgr2=0x7fbd70197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:03.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd701011e0 0x7fbd70197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6e59c640 1 -- 172.21.15.106:0/1008842943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd70101bd0 msgr2=0x7fbd7019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:03.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd70101bd0 0x7fbd7019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6e59c640 1 -- 172.21.15.106:0/1008842943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7019f640 con 0x7fbd70100820 2024-03-28T02:11:03.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 0x7fbd70197a20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400e9c0 tx=0x7fbd6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:03.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd5f7fe640 1 -- 172.21.15.106:0/1008842943 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400cd80 con 0x7fbd70100820 2024-03-28T02:11:03.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd5f7fe640 1 -- 172.21.15.106:0/1008842943 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd640035f0 con 0x7fbd70100820 2024-03-28T02:11:03.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd74859640 1 -- 172.21.15.106:0/1008842943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7019f930 con 0x7fbd70100820 2024-03-28T02:11:03.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd5f7fe640 1 -- 172.21.15.106:0/1008842943 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd64010690 con 0x7fbd70100820 2024-03-28T02:11:03.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.006+0000 7fbd74859640 1 -- 172.21.15.106:0/1008842943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7019fea0 con 0x7fbd70100820 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.009+0000 7fbd5f7fe640 1 -- 172.21.15.106:0/1008842943 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbd640037a0 con 0x7fbd70100820 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.009+0000 7fbd77440740 1 Processor -- start 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.009+0000 7fbd77440740 1 -- start start 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd77440740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea8537bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd77440740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537c130 0x55ea85380c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd77440740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea85383730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd77440740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea852e0c20 con 0x55ea853812a0 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd77440740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea85289a40 con 0x55ea8537b740 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd77440740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea852dccc0 con 0x55ea8537c130 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea85383730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea85383730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54940/0 (socket says 172.21.15.106:54940) 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/398260628 learned_addr learned my addr 172.21.15.106:0/398260628 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537c130 0x55ea85380c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea8537bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd5f7fe640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd580a2880 0x7fbd580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/398260628 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537c130 msgr2=0x55ea85380c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537c130 0x55ea85380c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/398260628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 msgr2=0x55ea8537bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea8537bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/398260628 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea852abf90 con 0x55ea853812a0 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea8537bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd580a2880 0x7fbd580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd5f7fe640 1 -- 172.21.15.106:0/1008842943 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd640cf130 con 0x7fbd70100820 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.010+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea85383730 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbd60000c00 tx=0x7fbd60009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc2d8ece3545dbd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd5cff9640 1 -- 172.21.15.106:0/398260628 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6002c980 con 0x55ea853812a0 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd580a2880 0x7fbd580a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55ea8537c990 tx=0x7fbd68004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:03.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd5cff9640 1 -- 172.21.15.106:0/398260628 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd6002cb20 con 0x55ea853812a0 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd5cff9640 1 -- 172.21.15.106:0/398260628 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd600046c0 con 0x55ea853812a0 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd77440740 1 -- 172.21.15.106:0/398260628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 msgr2=0x55ea85383730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd77440740 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea85383730 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbd60000c00 tx=0x7fbd60009560 comp rx=0 tx=0).stop 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd77440740 1 -- 172.21.15.106:0/398260628 shutdown_connections 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd77440740 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea85383730 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd77440740 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537c130 0x55ea85380c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd77440740 1 --2- 172.21.15.106:0/398260628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea8537bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd77440740 1 -- 172.21.15.106:0/398260628 >> 172.21.15.106:0/398260628 conn(0x55ea85343a60 msgr2=0x55ea853770b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd77440740 1 -- 172.21.15.106:0/398260628 shutdown_connections 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.011+0000 7fbd77440740 1 -- 172.21.15.106:0/398260628 wait complete. 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd77440740 1 -- start start 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd77440740 1 Processor -- start 2024-03-28T02:11:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd77440740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea85385b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd77440740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea853860d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd77440740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 0x55ea8538d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd77440740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea853841f0 con 0x55ea8537c130 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd77440740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea8529ce90 con 0x55ea8537b740 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd77440740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea85384370 con 0x55ea853812a0 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd77440740 10 client.0 _ll_register_callbacks cb 0x55ea84ff4ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea853860d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea853860d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54946/0 (socket says 172.21.15.106:54946) 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/3835449216 learned_addr learned my addr 172.21.15.106:0/3835449216 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea85385b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 0x55ea8538d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/3835449216 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 msgr2=0x55ea8538d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 0x55ea8538d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/3835449216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 msgr2=0x55ea85385b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea85385b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.012+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/3835449216 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea852a99d0 con 0x55ea8537c130 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea853860d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009900 tx=0x7fbd6003f910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f6d3f11bd0f444 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd55ffb640 1 -- 172.21.15.106:0/3835449216 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6002c980 con 0x55ea8537c130 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd55ffb640 1 -- 172.21.15.106:0/3835449216 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd6003fd00 con 0x55ea8537c130 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd77440740 1 -- 172.21.15.106:0/3835449216 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd6002cb20 con 0x55ea8537c130 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd55ffb640 1 -- 172.21.15.106:0/3835449216 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd6005a050 con 0x55ea8537c130 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd77440740 10 client.15510 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd77440740 1 -- 172.21.15.106:0/3835449216 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ea84ff52d0 con 0x55ea8537c130 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd55ffb640 1 -- 172.21.15.106:0/3835449216 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd6002cb20 con 0x55ea8537c130 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd55ffb640 1 -- 172.21.15.106:0/3835449216 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fbd600495b0 con 0x55ea8537c130 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd77440740 10 client.15510 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd77440740 10 client.15510 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T02:11:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd77440740 20 client.15510 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd77440740 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55ea85391270 0x55ea85396360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd77440740 4 client.15510 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd77440740 1 -- 172.21.15.106:0/3835449216 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea852ee160 con 0x55ea85391270 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.013+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55ea85391270 0x55ea85396360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.014+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55ea85391270 0x55ea85396360 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ef800185bbdf898 server_cookie=79e52d5882752a5c in_seq=0 out_seq=0 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.014+0000 7fbd55ffb640 10 client.15510 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.014+0000 7fbd55ffb640 1 -- 172.21.15.106:0/3835449216 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ea852ee160 con 0x55ea85391270 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.014+0000 7fbd55ffb640 10 client.15510 handle_command_reply: tid=0 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.014+0000 7fbd77440740 1 client.15510 shutdown 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.014+0000 7fbd77440740 1 -- 172.21.15.106:0/3835449216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 msgr2=0x55ea853860d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea853860d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009900 tx=0x7fbd6003f910 comp rx=0 tx=0).stop 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 1 -- 172.21.15.106:0/3835449216 shutdown_connections 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55ea85391270 0x55ea85396360 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 0x55ea8538d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea853860d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd55ffb640 0 client.15510 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:03.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 1 --2- 172.21.15.106:0/3835449216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea85385b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 1 -- 172.21.15.106:0/3835449216 >> 172.21.15.106:0/3835449216 conn(0x55ea85343a60 msgr2=0x55ea853762f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:03.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 1 -- 172.21.15.106:0/3835449216 shutdown_connections 2024-03-28T02:11:03.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 1 -- 172.21.15.106:0/3835449216 wait complete. 2024-03-28T02:11:03.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.015+0000 7fbd77440740 20 client.15510 trim_cache size 0 max 16384 2024-03-28T02:11:03.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd77440740 1 Processor -- start 2024-03-28T02:11:03.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd77440740 1 -- start start 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd77440740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea85375a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd77440740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea8538d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd77440740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 0x55ea85375fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd77440740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea85391700 con 0x55ea8537c130 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd77440740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea85391d60 con 0x55ea8537b740 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd77440740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea853923c0 con 0x55ea853812a0 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd6dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea8538d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd6dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea8538d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54962/0 (socket says 172.21.15.106:54962) 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/3379119236 learned_addr learned my addr 172.21.15.106:0/3379119236 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea85375a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.031+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 0x55ea85375fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/3379119236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 msgr2=0x55ea85375fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 0x55ea85375fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/3379119236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 msgr2=0x55ea85375a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea85375a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd6dd9b640 1 -- 172.21.15.106:0/3379119236 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea850bf6b0 con 0x55ea8537c130 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea85375a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea8538d0d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbd6002c170 tx=0x7fbd60009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e1c7e5528ff8fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd57fff640 1 -- 172.21.15.106:0/3379119236 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6002c980 con 0x55ea8537c130 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd57fff640 1 -- 172.21.15.106:0/3379119236 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd6002cb20 con 0x55ea8537c130 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd57fff640 1 -- 172.21.15.106:0/3379119236 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd60034650 con 0x55ea8537c130 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd77440740 1 -- 172.21.15.106:0/3379119236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 msgr2=0x55ea8538d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd77440740 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea8538d0d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbd6002c170 tx=0x7fbd60009650 comp rx=0 tx=0).stop 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd77440740 1 -- 172.21.15.106:0/3379119236 shutdown_connections 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd77440740 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea853812a0 0x55ea85375fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd77440740 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea8537c130 0x55ea8538d0d0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd77440740 1 --2- 172.21.15.106:0/3379119236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537b740 0x55ea85375a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd77440740 1 -- 172.21.15.106:0/3379119236 >> 172.21.15.106:0/3379119236 conn(0x55ea85343a60 msgr2=0x55ea85385b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd77440740 1 -- 172.21.15.106:0/3379119236 shutdown_connections 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.032+0000 7fbd77440740 1 -- 172.21.15.106:0/3379119236 wait complete. 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd77440740 1 -- start start 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd77440740 1 Processor -- start 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd77440740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537b740 0x55ea85388ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd77440740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537c130 0x55ea853893e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd77440740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea8538c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd77440740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea85393080 con 0x55ea853812a0 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd77440740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea85392a20 con 0x55ea8537c130 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd77440740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea853936e0 con 0x55ea8537b740 2024-03-28T02:11:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd77440740 10 client.0 _ll_register_callbacks cb 0x7fbd68009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6ed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea8538c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6ed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea8538c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54966/0 (socket says 172.21.15.106:54966) 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6ed9d640 1 -- 172.21.15.106:0/1399597050 learned_addr learned my addr 172.21.15.106:0/1399597050 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537b740 0x55ea85388ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6dd9b640 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537c130 0x55ea853893e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6ed9d640 1 -- 172.21.15.106:0/1399597050 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537b740 msgr2=0x55ea85388ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537b740 0x55ea85388ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6ed9d640 1 -- 172.21.15.106:0/1399597050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537c130 msgr2=0x55ea853893e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537c130 0x55ea853893e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.033+0000 7fbd6ed9d640 1 -- 172.21.15.106:0/1399597050 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea8538c9f0 con 0x55ea853812a0 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd6ed9d640 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea8538c400 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbd680209a0 tx=0x7fbd68004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf42b613acaf341 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd55ffb640 1 -- 172.21.15.106:0/1399597050 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6802f060 con 0x55ea853812a0 2024-03-28T02:11:03.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd55ffb640 1 -- 172.21.15.106:0/1399597050 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd6801edc0 con 0x55ea853812a0 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd77440740 1 -- 172.21.15.106:0/1399597050 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd6002cb20 con 0x55ea853812a0 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd55ffb640 1 -- 172.21.15.106:0/1399597050 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd6802a450 con 0x55ea853812a0 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd77440740 10 client.15516 fetch_fsmap learned FSMap version 28 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd77440740 1 -- 172.21.15.106:0/1399597050 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ea853edb70 con 0x55ea853812a0 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd55ffb640 1 -- 172.21.15.106:0/1399597050 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd6002cb20 con 0x55ea853812a0 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd55ffb640 1 -- 172.21.15.106:0/1399597050 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fbd68010730 con 0x55ea853812a0 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd77440740 10 client.15516 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd77440740 10 client.15516 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd77440740 20 client.15516 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd77440740 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55ea853f2340 0x55ea853edeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd77440740 4 client.15516 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd77440740 1 -- 172.21.15.106:0/1399597050 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ea853773a0 con 0x55ea853f2340 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.034+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55ea853f2340 0x55ea853edeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:03.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd6e59c640 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55ea853f2340 0x55ea853edeb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d40fbabb5922361 server_cookie=75e46265f26064a8 in_seq=0 out_seq=0 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd55ffb640 10 client.15516 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd55ffb640 1 -- 172.21.15.106:0/1399597050 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55ea853773a0 con 0x55ea853f2340 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd55ffb640 10 client.15516 handle_command_reply: tid=0 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 client.15516 shutdown 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 -- 172.21.15.106:0/1399597050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 msgr2=0x55ea8538c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea8538c400 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbd680209a0 tx=0x7fbd68004ab0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 -- 172.21.15.106:0/1399597050 shutdown_connections 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55ea853f2340 0x55ea853edeb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ea853812a0 0x55ea8538c400 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea8537c130 0x55ea853893e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 --2- 172.21.15.106:0/1399597050 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55ea8537b740 0x55ea85388ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd55ffb640 0 client.15516 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:03.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 -- 172.21.15.106:0/1399597050 >> 172.21.15.106:0/1399597050 conn(0x55ea85343a60 msgr2=0x55ea85384710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 -- 172.21.15.106:0/1399597050 shutdown_connections 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 1 -- 172.21.15.106:0/1399597050 wait complete. 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.035+0000 7fbd77440740 20 client.15516 trim_cache size 0 max 16384 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_fsid": "d222a03c-eca6-11ee-95d0-87774f69a715", 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stdout: "whoami": 0, 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 24509, 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stdout: "want_state": "up:active", 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stdout: "state": "up:active", 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs", 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stdout: "rank_uptime": 133.59034592899999, 2024-03-28T02:11:03.036 INFO:teuthology.orchestra.run.smithi106.stdout: "mdsmap_epoch": 28, 2024-03-28T02:11:03.037 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch": 93, 2024-03-28T02:11:03.037 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T02:11:03.037 INFO:teuthology.orchestra.run.smithi106.stdout: "uptime": 140.285238978 2024-03-28T02:11:03.037 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:11:03.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 -- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd580a2880 msgr2=0x7fbd580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd580a2880 0x7fbd580a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55ea8537c990 tx=0x7fbd68004a40 comp rx=0 tx=0).stop 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 -- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 msgr2=0x7fbd70197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 0x7fbd70197a20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400e9c0 tx=0x7fbd6400ee80 comp rx=0 tx=0).stop 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 -- 172.21.15.106:0/1008842943 shutdown_connections 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fbd580a2880 0x7fbd580a4d40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd70101bd0 0x7fbd7019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd701011e0 0x7fbd70197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 --2- 172.21.15.106:0/1008842943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd70100820 0x7fbd70197a20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 -- 172.21.15.106:0/1008842943 >> 172.21.15.106:0/1008842943 conn(0x7fbd700fb2b0 msgr2=0x7fbd70104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 -- 172.21.15.106:0/1008842943 shutdown_connections 2024-03-28T02:11:03.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:03.036+0000 7fbd74859640 1 -- 172.21.15.106:0/1008842943 wait complete. 2024-03-28T02:11:03.039 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T02:11:03.053 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T02:11:03.053 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T02:11:03.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:03.053 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list 2024-03-28T02:11:03.068 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-28T02:11:03.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:03.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T02:11:03.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:03.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link delete ceph-brx 2024-03-28T02:11:03.231 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-28T02:11:03.231 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T02:11:03.231 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T02:11:03.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T02:11:03.475 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:11:03.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:03 smithi153 ceph-mon[36228]: pgmap v363: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-03-28T02:11:03.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:03 smithi106 ceph-mon[31965]: pgmap v363: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-03-28T02:11:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:03 smithi178 ceph-mon[36521]: pgmap v363: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.135+0000 7ff7209c9700 1 -- 172.21.15.106:0/3524521477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 msgr2=0x7ff71c0747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.135+0000 7ff7209c9700 1 --2- 172.21.15.106:0/3524521477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 0x7ff71c0747a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff714009990 tx=0x7ff714009ca0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.135+0000 7ff7209c9700 1 -- 172.21.15.106:0/3524521477 shutdown_connections 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.135+0000 7ff7209c9700 1 --2- 172.21.15.106:0/3524521477 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff71c10eb90 0x7ff71c110f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.135+0000 7ff7209c9700 1 --2- 172.21.15.106:0/3524521477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 0x7ff71c0747a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.135+0000 7ff7209c9700 1 --2- 172.21.15.106:0/3524521477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c074ce0 0x7ff71c10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.135+0000 7ff7209c9700 1 -- 172.21.15.106:0/3524521477 >> 172.21.15.106:0/3524521477 conn(0x7ff71c0792c0 msgr2=0x7ff71c0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.135+0000 7ff7209c9700 1 -- 172.21.15.106:0/3524521477 shutdown_connections 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.135+0000 7ff7209c9700 1 -- 172.21.15.106:0/3524521477 wait complete. 2024-03-28T02:11:05.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff7209c9700 1 Processor -- start 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff7209c9700 1 -- start start 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff7209c9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 0x7ff71c1a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff7209c9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c074ce0 0x7ff71c1a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff7209c9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff71c10eb90 0x7ff71c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff7209c9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c1146b0 con 0x7ff71c0743c0 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff7209c9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c114530 con 0x7ff71c074ce0 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff7209c9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c114830 con 0x7ff71c10eb90 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff71a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 0x7ff71c1a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff71a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 0x7ff71c1a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54980/0 (socket says 172.21.15.106:54980) 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.136+0000 7ff71a59c700 1 -- 172.21.15.106:0/1885103154 learned_addr learned my addr 172.21.15.106:0/1885103154 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff719d9b700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c074ce0 0x7ff71c1a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff71ad9d700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff71c10eb90 0x7ff71c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff71a59c700 1 -- 172.21.15.106:0/1885103154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff71c10eb90 msgr2=0x7ff71c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:05.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff71a59c700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff71c10eb90 0x7ff71c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff71a59c700 1 -- 172.21.15.106:0/1885103154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c074ce0 msgr2=0x7ff71c1a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff71a59c700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c074ce0 0x7ff71c1a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff71a59c700 1 -- 172.21.15.106:0/1885103154 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c19b0d0 con 0x7ff71c0743c0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff71a59c700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 0x7ff71c1a07b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff714005910 tx=0x7ff71400bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff70b7fe700 1 -- 172.21.15.106:0/1885103154 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff714027070 con 0x7ff71c0743c0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c19b360 con 0x7ff71c0743c0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff70b7fe700 1 -- 172.21.15.106:0/1885103154 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff714004650 con 0x7ff71c0743c0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff70b7fe700 1 -- 172.21.15.106:0/1885103154 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71400f460 con 0x7ff71c0743c0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.137+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c1a7680 con 0x7ff71c0743c0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.138+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c068e50 con 0x7ff71c0743c0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.139+0000 7ff70b7fe700 1 -- 172.21.15.106:0/1885103154 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff71402a070 con 0x7ff71c0743c0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.140+0000 7ff70b7fe700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff7040a2890 0x7ff7040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.140+0000 7ff719d9b700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff7040a2890 0x7ff7040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:05.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.140+0000 7ff70b7fe700 1 -- 172.21.15.106:0/1885103154 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff7140d00f0 con 0x7ff71c0743c0 2024-03-28T02:11:05.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.140+0000 7ff719d9b700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff7040a2890 0x7ff7040a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff71c101c70 tx=0x7ff71000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:05.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.143+0000 7ff70b7fe700 1 -- 172.21.15.106:0/1885103154 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff714031070 con 0x7ff71c0743c0 2024-03-28T02:11:05.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.390+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff71c02da30 con 0x7ff71c0743c0 2024-03-28T02:11:05.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.391+0000 7ff70b7fe700 1 -- 172.21.15.106:0/1885103154 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff714013070 con 0x7ff71c0743c0 2024-03-28T02:11:05.384 INFO:teuthology.orchestra.run.smithi106.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T02:11:05.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff7040a2890 msgr2=0x7ff7040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:05.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff7040a2890 0x7ff7040a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff71c101c70 tx=0x7ff71000b040 comp rx=0 tx=0).stop 2024-03-28T02:11:05.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 msgr2=0x7ff71c1a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:05.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 0x7ff71c1a07b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff714005910 tx=0x7ff71400bab0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 shutdown_connections 2024-03-28T02:11:05.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff71c10eb90 0x7ff71c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7ff7040a2890 0x7ff7040a4d50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff71c0743c0 0x7ff71c1a07b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 --2- 172.21.15.106:0/1885103154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c074ce0 0x7ff71c1a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:05.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 >> 172.21.15.106:0/1885103154 conn(0x7ff71c0792c0 msgr2=0x7ff71c0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:05.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 shutdown_connections 2024-03-28T02:11:05.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:05.393+0000 7ff7209c9700 1 -- 172.21.15.106:0/1885103154 wait complete. 2024-03-28T02:11:05.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:05 smithi153 ceph-mon[36228]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T02:11:05.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:05 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1885103154' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T02:11:05.758 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:05 smithi106 ceph-mon[31965]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T02:11:05.758 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:05 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1885103154' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T02:11:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:05 smithi178 ceph-mon[36521]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T02:11:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:05 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1885103154' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T02:11:06.106 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T02:11:06.106 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T02:11:06.106 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi106.front.sepia.ceph.com 2024-03-28T02:11:06.106 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T02:11:06.106 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T02:11:06.106 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T02:11:06.106 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T02:11:06.106 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T02:11:06.106 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T02:11:06.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.106 DEBUG:teuthology.orchestra.run.smithi106:> ip addr 2024-03-28T02:11:06.122 INFO:teuthology.orchestra.run.smithi106.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T02:11:06.122 INFO:teuthology.orchestra.run.smithi106.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T02:11:06.122 INFO:teuthology.orchestra.run.smithi106.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T02:11:06.122 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-03-28T02:11:06.122 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 ::1/128 scope host 2024-03-28T02:11:06.122 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-03-28T02:11:06.122 INFO:teuthology.orchestra.run.smithi106.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:d8:a1:2a brd ff:ff:ff:ff:ff:ff 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: altname enp7s0f0 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:d8:a1:2b brd ff:ff:ff:ff:ff:ff 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: altname enp7s0f1 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:8f:ce:6c brd ff:ff:ff:ff:ff:ff 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:8f:ce:6d brd ff:ff:ff:ff:ff:ff 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: inet 172.21.15.106/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft 41968sec preferred_lft 41968sec 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 fe80::ec4:7aff:fe8f:ce6d/64 scope link 2024-03-28T02:11:06.123 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-03-28T02:11:06.124 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T02:11:06.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.124 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:11:06.124 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:11:06.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link add name ceph-brx type bridge 2024-03-28T02:11:06.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip addr flush dev ceph-brx 2024-03-28T02:11:06.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set ceph-brx up 2024-03-28T02:11:06.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T02:11:06.124 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:11:06.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.234 DEBUG:teuthology.orchestra.run.smithi106:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T02:11:06.259 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-03-28T02:11:06.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.261 DEBUG:teuthology.orchestra.run.smithi106:> route 2024-03-28T02:11:06.317 INFO:teuthology.orchestra.run.smithi106.stdout:Kernel IP routing table 2024-03-28T02:11:06.317 INFO:teuthology.orchestra.run.smithi106.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T02:11:06.317 INFO:teuthology.orchestra.run.smithi106.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T02:11:06.318 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T02:11:06.318 INFO:teuthology.orchestra.run.smithi106.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T02:11:06.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.318 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:11:06.318 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:11:06.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T02:11:06.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T02:11:06.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T02:11:06.318 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:11:06.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.415 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list 2024-03-28T02:11:06.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.429 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list-id 2024-03-28T02:11:06.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.483 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:11:06.483 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:11:06.483 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T02:11:06.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T02:11:06.484 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:11:06.566 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T02:11:06.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.566 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:11:06.566 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:11:06.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T02:11:06.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T02:11:06.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T02:11:06.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T02:11:06.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T02:11:06.566 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:11:06.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.693 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-03-28T02:11:06.693 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-03-28T02:11:06.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set brx.0 up 2024-03-28T02:11:06.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T02:11:06.693 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-03-28T02:11:06.735 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T02:11:06.735 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T02:11:06.787 INFO:teuthology.orchestra.run.smithi106.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T02:11:06.787 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T02:11:06.787 DEBUG:teuthology.orchestra.run.smithi106:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T02:11:06.842 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T02:11:06.842 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T02:11:06.843 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/018493ee-3c1f-4528-b707-618afc45dfef 2024-03-28T02:11:06.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:06.843 DEBUG:teuthology.orchestra.run.smithi106:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/018493ee-3c1f-4528-b707-618afc45dfef /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-03-28T02:11:06.927 INFO:teuthology.orchestra.run.smithi106.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-03-28T02:11:06.927 INFO:teuthology.orchestra.run.smithi106.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-03-28T02:11:06.927 INFO:teuthology.orchestra.run.smithi106.stdout:invalid new device string format 2024-03-28T02:11:07.047 INFO:teuthology.orchestra.run.smithi106.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-03-28T02:11:07.047 INFO:teuthology.orchestra.run.smithi106.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-03-28T02:11:07.047 INFO:teuthology.orchestra.run.smithi106.stdout:invalid new device string format 2024-03-28T02:11:07.047 INFO:teuthology.orchestra.run.smithi106.stdout:mount.ceph: resolved to: "172.21.15.106:3300,172.21.15.153:3300,172.21.15.178:3300" 2024-03-28T02:11:07.047 INFO:teuthology.orchestra.run.smithi106.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.106:3300,172.21.15.153:3300,172.21.15.178:3300:/volumes/qa/sv_1/018493ee-3c1f-4528-b707-618afc45dfef 2024-03-28T02:11:07.047 INFO:teuthology.orchestra.run.smithi106.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=d222a03c-eca6-11ee-95d0-87774f69a715" will pass to kernel 2024-03-28T02:11:07.052 INFO:teuthology.orchestra.run.smithi106.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T02:11:07.052 INFO:teuthology.orchestra.run.smithi106.stdout: You just mounted a file system that supports labels which does not 2024-03-28T02:11:07.052 INFO:teuthology.orchestra.run.smithi106.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T02:11:07.052 INFO:teuthology.orchestra.run.smithi106.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T02:11:07.052 INFO:teuthology.orchestra.run.smithi106.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T02:11:07.052 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T02:11:07.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:07.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T02:11:07.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:07.086 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T02:11:07.143 INFO:teuthology.orchestra.run.smithi106.stdout:client25373 2024-03-28T02:11:07.144 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:11:07.144 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T02:11:07.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:07.200 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T02:11:07.256 INFO:teuthology.orchestra.run.smithi106.stdout:client25373 2024-03-28T02:11:07.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:07.257 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T02:11:07.313 INFO:teuthology.orchestra.run.smithi106.stdout:client25373 2024-03-28T02:11:07.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:11:07.313 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d222a03c-eca6-11ee-95d0-87774f69a715.client25373/status') 2024-03-28T02:11:07.379 INFO:teuthology.orchestra.run.smithi106.stdout:instance: client.25373 (3)172.21.15.106:0/1507897534 2024-03-28T02:11:07.379 INFO:teuthology.orchestra.run.smithi106.stdout:blocklisted: false 2024-03-28T02:11:07.379 INFO:teuthology.orchestra.run.smithi106.stderr:0+1 records in 2024-03-28T02:11:07.379 INFO:teuthology.orchestra.run.smithi106.stderr:0+1 records out 2024-03-28T02:11:07.379 INFO:teuthology.orchestra.run.smithi106.stderr:72 bytes copied, 1.8879e-05 s, 3.8 MB/s 2024-03-28T02:11:07.381 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T02:11:07.390 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T02:11:07.390 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:11:07.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-28T02:11:07.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.535+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/3705656492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb981016d0 msgr2=0x7fcb98101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:07.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.535+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/3705656492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb981016d0 0x7fcb98101af0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009900 tx=0x7fcb8802bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:07.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.535+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/3705656492 shutdown_connections 2024-03-28T02:11:07.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.535+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/3705656492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98103280 0x7fcb98109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:07.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.535+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/3705656492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb981028c0 0x7fcb98102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:07.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.535+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/3705656492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb981016d0 0x7fcb98101af0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:07.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.535+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/3705656492 >> 172.21.15.106:0/3705656492 conn(0x7fcb980fcc60 msgr2=0x7fcb980ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:07.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.535+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/3705656492 shutdown_connections 2024-03-28T02:11:07.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.535+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/3705656492 wait complete. 2024-03-28T02:11:07.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.536+0000 7fcb9d8e8640 1 Processor -- start 2024-03-28T02:11:07.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.536+0000 7fcb9d8e8640 1 -- start start 2024-03-28T02:11:07.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.536+0000 7fcb9d8e8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb981016d0 0x7fcb98197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:07.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.536+0000 7fcb9d8e8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb981028c0 0x7fcb98197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.536+0000 7fcb9d8e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98103280 0x7fcb9819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.536+0000 7fcb9d8e8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9810d320 con 0x7fcb981016d0 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.536+0000 7fcb9d8e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9810d1a0 con 0x7fcb98103280 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.536+0000 7fcb9d8e8640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9810d4a0 con 0x7fcb981028c0 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.536+0000 7fcb96ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb981016d0 0x7fcb98197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb96ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb981016d0 0x7fcb98197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37106/0 (socket says 172.21.15.106:37106) 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb96ffd640 1 -- 172.21.15.106:0/531924699 learned_addr learned my addr 172.21.15.106:0/531924699 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb977fe640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98103280 0x7fcb9819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb967fc640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb981028c0 0x7fcb98197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb96ffd640 1 -- 172.21.15.106:0/531924699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb981028c0 msgr2=0x7fcb98197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb96ffd640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb981028c0 0x7fcb98197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb96ffd640 1 -- 172.21.15.106:0/531924699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98103280 msgr2=0x7fcb9819eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb96ffd640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98103280 0x7fcb9819eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:07.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb96ffd640 1 -- 172.21.15.106:0/531924699 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9819f550 con 0x7fcb981016d0 2024-03-28T02:11:07.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb96ffd640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb981016d0 0x7fcb98197930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcb88000c00 tx=0x7fcb8802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:07.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb9c8e6640 1 -- 172.21.15.106:0/531924699 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88041070 con 0x7fcb981016d0 2024-03-28T02:11:07.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9819f7e0 con 0x7fcb981016d0 2024-03-28T02:11:07.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb9c8e6640 1 -- 172.21.15.106:0/531924699 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb88004450 con 0x7fcb981016d0 2024-03-28T02:11:07.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb9c8e6640 1 -- 172.21.15.106:0/531924699 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88034440 con 0x7fcb981016d0 2024-03-28T02:11:07.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.537+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9819fd20 con 0x7fcb981016d0 2024-03-28T02:11:07.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.538+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb98108990 con 0x7fcb981016d0 2024-03-28T02:11:07.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.539+0000 7fcb9c8e6640 1 -- 172.21.15.106:0/531924699 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcb88044070 con 0x7fcb981016d0 2024-03-28T02:11:07.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.540+0000 7fcb9c8e6640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcb800a28a0 0x7fcb800a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:07.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.540+0000 7fcb967fc640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcb800a28a0 0x7fcb800a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:07.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.540+0000 7fcb9c8e6640 1 -- 172.21.15.106:0/531924699 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fcb880f08c0 con 0x7fcb981016d0 2024-03-28T02:11:07.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.540+0000 7fcb967fc640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcb800a28a0 0x7fcb800a4d60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0095f0 tx=0x7fcb8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:07.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.544+0000 7fcb9c8e6640 1 -- 172.21.15.106:0/531924699 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb880bd000 con 0x7fcb981016d0 2024-03-28T02:11:07.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:07 smithi153 ceph-mon[36228]: pgmap v365: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:11:07.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:07.759+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7fcb9802d070 con 0x7fcb981016d0 2024-03-28T02:11:07.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:07 smithi106 ceph-mon[31965]: pgmap v365: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:11:07.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:07 smithi178 ceph-mon[36521]: pgmap v365: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T02:11:08.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.480+0000 7fcb9c8e6640 1 -- 172.21.15.106:0/531924699 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) v1 ==== 117+0+0 (secure 0 0 0) 0x7fcb880c10a0 con 0x7fcb981016d0 2024-03-28T02:11:08.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcb800a28a0 msgr2=0x7fcb800a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:08.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcb800a28a0 0x7fcb800a4d60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0095f0 tx=0x7fcb8c005f90 comp rx=0 tx=0).stop 2024-03-28T02:11:08.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb981016d0 msgr2=0x7fcb98197930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:08.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb981016d0 0x7fcb98197930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcb88000c00 tx=0x7fcb8802c640 comp rx=0 tx=0).stop 2024-03-28T02:11:08.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 shutdown_connections 2024-03-28T02:11:08.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fcb800a28a0 0x7fcb800a4d60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98103280 0x7fcb9819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb981028c0 0x7fcb98197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 --2- 172.21.15.106:0/531924699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb981016d0 0x7fcb98197930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 >> 172.21.15.106:0/531924699 conn(0x7fcb980fcc60 msgr2=0x7fcb98105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:08.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 shutdown_connections 2024-03-28T02:11:08.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.484+0000 7fcb9d8e8640 1 -- 172.21.15.106:0/531924699 wait complete. 2024-03-28T02:11:08.490 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T02:11:08.501 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T02:11:08.524 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T02:11:08.524 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:11:08.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 -- 172.21.15.106:0/525998261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f0102180 msgr2=0x7fe2f0108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 --2- 172.21.15.106:0/525998261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f0102180 0x7fe2f0108a10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009be0 tx=0x7fe2e002bef0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 -- 172.21.15.106:0/525998261 shutdown_connections 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 --2- 172.21.15.106:0/525998261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f0102180 0x7fe2f0108a10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 --2- 172.21.15.106:0/525998261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2f01017c0 0x7fe2f0101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 --2- 172.21.15.106:0/525998261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2f01005d0 0x7fe2f01009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 -- 172.21.15.106:0/525998261 >> 172.21.15.106:0/525998261 conn(0x7fe2f00fbb60 msgr2=0x7fe2f00fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 -- 172.21.15.106:0/525998261 shutdown_connections 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 -- 172.21.15.106:0/525998261 wait complete. 2024-03-28T02:11:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 Processor -- start 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.639+0000 7fe2f5589640 1 -- start start 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2f5589640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f01005d0 0x7fe2f01978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2f5589640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2f01017c0 0x7fe2f0197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2f5589640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2f0102180 0x7fe2f019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2f5589640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f010c230 con 0x7fe2f01017c0 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2f5589640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f010c0b0 con 0x7fe2f01005d0 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2f5589640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f010c3b0 con 0x7fe2f0102180 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2ee7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2f01017c0 0x7fe2f0197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2eeffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f01005d0 0x7fe2f01978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2eeffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f01005d0 0x7fe2f01978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:51306/0 (socket says 172.21.15.106:51306) 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2eeffd640 1 -- 172.21.15.106:0/4221354857 learned_addr learned my addr 172.21.15.106:0/4221354857 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2ef7fe640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2f0102180 0x7fe2f019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:08.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2eeffd640 1 -- 172.21.15.106:0/4221354857 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2f0102180 msgr2=0x7fe2f019ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2eeffd640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2f0102180 0x7fe2f019ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2eeffd640 1 -- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2f01017c0 msgr2=0x7fe2f0197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2eeffd640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2f01017c0 0x7fe2f0197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2eeffd640 1 -- 172.21.15.106:0/4221354857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f019f4c0 con 0x7fe2f01005d0 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2ee7fc640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2f01017c0 0x7fe2f0197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2eeffd640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f01005d0 0x7fe2f01978a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800e9f0 tx=0x7fe2e800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2e7fff640 1 -- 172.21.15.106:0/4221354857 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800ccc0 con 0x7fe2f01005d0 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2e7fff640 1 -- 172.21.15.106:0/4221354857 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2e80035f0 con 0x7fe2f01005d0 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f019f7b0 con 0x7fe2f01005d0 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2e7fff640 1 -- 172.21.15.106:0/4221354857 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800ce60 con 0x7fe2f01005d0 2024-03-28T02:11:08.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.640+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f019fd00 con 0x7fe2f01005d0 2024-03-28T02:11:08.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.641+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f019ffc0 con 0x7fe2f01005d0 2024-03-28T02:11:08.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.642+0000 7fe2e7fff640 1 -- 172.21.15.106:0/4221354857 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe2e80037a0 con 0x7fe2f01005d0 2024-03-28T02:11:08.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.643+0000 7fe2e7fff640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2dc0a28a0 0x7fe2dc0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:08.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.643+0000 7fe2ee7fc640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2dc0a28a0 0x7fe2dc0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:08.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.643+0000 7fe2e7fff640 1 -- 172.21.15.106:0/4221354857 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fe2e80cf060 con 0x7fe2f01005d0 2024-03-28T02:11:08.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.643+0000 7fe2ee7fc640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2dc0a28a0 0x7fe2dc0a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0198db0 tx=0x7fe2d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:08.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.648+0000 7fe2e7fff640 1 -- 172.21.15.106:0/4221354857 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe2e809bd60 con 0x7fe2f01005d0 2024-03-28T02:11:08.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:08 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/531924699' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T02:11:08.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:08 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/531924699' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T02:11:08.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:08.873+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe2f01a0770 con 0x7fe2f01005d0 2024-03-28T02:11:08.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:08 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/531924699' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T02:11:09.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.494+0000 7fe2e7fff640 1 -- 172.21.15.106:0/4221354857 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 19) v1 ==== 137308+0+0 (secure 0 0 0) 0x7fe2e8097cd0 con 0x7fe2f01005d0 2024-03-28T02:11:09.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.495+0000 7fe2e7fff640 1 -- 172.21.15.106:0/4221354857 <== mon.2 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe2e80771c0 con 0x7fe2f01005d0 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2dc0a28a0 msgr2=0x7fe2dc0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2dc0a28a0 0x7fe2dc0a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0198db0 tx=0x7fe2d8005f90 comp rx=0 tx=0).stop 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f01005d0 msgr2=0x7fe2f01978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f01005d0 0x7fe2f01978a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800e9f0 tx=0x7fe2e800eeb0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 shutdown_connections 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2f0102180 0x7fe2f019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2f01017c0 0x7fe2f0197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fe2dc0a28a0 0x7fe2dc0a4d60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 --2- 172.21.15.106:0/4221354857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f01005d0 0x7fe2f01978a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.498+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 >> 172.21.15.106:0/4221354857 conn(0x7fe2f00fbb60 msgr2=0x7fe2f0069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:09.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.499+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 shutdown_connections 2024-03-28T02:11:09.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.499+0000 7fe2f5589640 1 -- 172.21.15.106:0/4221354857 wait complete. 2024-03-28T02:11:09.505 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-28T02:11:09.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 -- 172.21.15.106:0/3663787244 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218102140 msgr2=0x7f7218104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:09.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3663787244 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218102140 0x7f7218104530 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7208009900 tx=0x7f720802bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:09.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 -- 172.21.15.106:0/3663787244 shutdown_connections 2024-03-28T02:11:09.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3663787244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72181073a0 0x7f72181097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3663787244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7218104a70 0x7f7218106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3663787244 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218102140 0x7f7218104530 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 -- 172.21.15.106:0/3663787244 >> 172.21.15.106:0/3663787244 conn(0x7f72180fbb30 msgr2=0x7f72180fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 -- 172.21.15.106:0/3663787244 shutdown_connections 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 -- 172.21.15.106:0/3663787244 wait complete. 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.621+0000 7f721dc8c640 1 Processor -- start 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f721dc8c640 1 -- start start 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f721dc8c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7218102140 0x7f7218197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f721dc8c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218104a70 0x7f7218197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f721dc8c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72181073a0 0x7f721819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f721dc8c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721810d360 con 0x7f7218104a70 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f721dc8c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721810d1e0 con 0x7f7218102140 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f721dc8c640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721810d4e0 con 0x7f72181073a0 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7216ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218104a70 0x7f7218197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7216ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218104a70 0x7f7218197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37178/0 (socket says 172.21.15.106:37178) 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f72177fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7218102140 0x7f7218197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7216ffd640 1 -- 172.21.15.106:0/3628939964 learned_addr learned my addr 172.21.15.106:0/3628939964 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7217fff640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72181073a0 0x7f721819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7216ffd640 1 -- 172.21.15.106:0/3628939964 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72181073a0 msgr2=0x7f721819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7216ffd640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72181073a0 0x7f721819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7216ffd640 1 -- 172.21.15.106:0/3628939964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7218102140 msgr2=0x7f7218197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7216ffd640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7218102140 0x7f7218197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7216ffd640 1 -- 172.21.15.106:0/3628939964 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721819f590 con 0x7f7218104a70 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.622+0000 7f7216ffd640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218104a70 0x7f7218197eb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f720c00e9f0 tx=0x7f720c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.623+0000 7f7214ff9640 1 -- 172.21.15.106:0/3628939964 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c00ccc0 con 0x7f7218104a70 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.623+0000 7f7214ff9640 1 -- 172.21.15.106:0/3628939964 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f720c005500 con 0x7f7218104a70 2024-03-28T02:11:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.623+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721819f880 con 0x7f7218104a70 2024-03-28T02:11:09.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.623+0000 7f7214ff9640 1 -- 172.21.15.106:0/3628939964 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c010690 con 0x7f7218104a70 2024-03-28T02:11:09.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.623+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721819fd80 con 0x7f7218104a70 2024-03-28T02:11:09.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.624+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72180fd620 con 0x7f7218104a70 2024-03-28T02:11:09.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.625+0000 7f7214ff9640 1 -- 172.21.15.106:0/3628939964 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 19) v1 ==== 137308+0+0 (secure 0 0 0) 0x7f720c0046c0 con 0x7f7218104a70 2024-03-28T02:11:09.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.625+0000 7f7214ff9640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f72000a28f0 0x7f72000a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:09.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.625+0000 7f72177fe640 1 -- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f72000a28f0 msgr2=0x7f72000a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/3501119323 2024-03-28T02:11:09.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.625+0000 7f7214ff9640 1 -- 172.21.15.106:0/3628939964 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f720c0cf1c0 con 0x7f7218104a70 2024-03-28T02:11:09.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.625+0000 7f72177fe640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f72000a28f0 0x7f72000a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T02:11:09.622 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.630+0000 7f7214ff9640 1 -- 172.21.15.106:0/3628939964 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f720c09bec0 con 0x7f7218104a70 2024-03-28T02:11:09.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.814+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f72180622a0 con 0x7f7218104a70 2024-03-28T02:11:09.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.817+0000 7f7214ff9640 1 -- 172.21.15.106:0/3628939964 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7f720c09b7c0 con 0x7f7218104a70 2024-03-28T02:11:09.813 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f72000a28f0 msgr2=0x7f72000a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f72000a28f0 0x7f72000a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218104a70 msgr2=0x7f7218197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218104a70 0x7f7218197eb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f720c00e9f0 tx=0x7f720c00eeb0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 shutdown_connections 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72181073a0 0x7f721819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7218104a70 0x7f7218197eb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f72000a28f0 0x7f72000a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 --2- 172.21.15.106:0/3628939964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7218102140 0x7f7218197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 >> 172.21.15.106:0/3628939964 conn(0x7f72180fbb30 msgr2=0x7f72181029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 shutdown_connections 2024-03-28T02:11:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.820+0000 7f721dc8c640 1 -- 172.21.15.106:0/3628939964 wait complete. 2024-03-28T02:11:09.823 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: ignoring --setuser ceph since I am not root 2024-03-28T02:11:09.824 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: ignoring --setgroup ceph since I am not root 2024-03-28T02:11:09.824 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:09 smithi106 ceph-mon[31965]: pgmap v366: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:11:09.824 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:09 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/531924699' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T02:11:09.824 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:09 smithi106 ceph-mon[31965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T02:11:09.824 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:09 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4221354857' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T02:11:09.824 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:09 smithi106 ceph-mon[31965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T02:11:09.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-28T02:11:09.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:09 smithi178 ceph-mon[36521]: pgmap v366: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:11:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:09 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/531924699' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T02:11:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:09 smithi178 ceph-mon[36521]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T02:11:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:09 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4221354857' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T02:11:09.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:09 smithi178 ceph-mon[36521]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T02:11:09.872 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:09 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: ignoring --setuser ceph since I am not root 2024-03-28T02:11:09.872 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:09 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: ignoring --setgroup ceph since I am not root 2024-03-28T02:11:09.872 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:09 smithi178 ceph-mgr[38388]: -- 172.21.15.178:0/2889210839 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563a5fbe85a0 con 0x563a5fbbcc00 2024-03-28T02:11:09.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.937+0000 7fa49d39d640 1 -- 172.21.15.106:0/1194926557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 msgr2=0x7fa498106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:09.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.937+0000 7fa49d39d640 1 --2- 172.21.15.106:0/1194926557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 0x7fa498106cf0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009900 tx=0x7fa48c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:09.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.937+0000 7fa49d39d640 1 -- 172.21.15.106:0/1194926557 shutdown_connections 2024-03-28T02:11:09.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.937+0000 7fa49d39d640 1 --2- 172.21.15.106:0/1194926557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa498107230 0x7fa498109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.937+0000 7fa49d39d640 1 --2- 172.21.15.106:0/1194926557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 0x7fa498106cf0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.937+0000 7fa49d39d640 1 --2- 172.21.15.106:0/1194926557 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa498101fd0 0x7fa4981043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.937+0000 7fa49d39d640 1 -- 172.21.15.106:0/1194926557 >> 172.21.15.106:0/1194926557 conn(0x7fa4980fb980 msgr2=0x7fa4980fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:09.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.937+0000 7fa49d39d640 1 -- 172.21.15.106:0/1194926557 shutdown_connections 2024-03-28T02:11:09.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.937+0000 7fa49d39d640 1 -- 172.21.15.106:0/1194926557 wait complete. 2024-03-28T02:11:09.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa49d39d640 1 Processor -- start 2024-03-28T02:11:09.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa49d39d640 1 -- start start 2024-03-28T02:11:09.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa49d39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa498101fd0 0x7fa498101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:09.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa49d39d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 0x7fa4980fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:09.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa49d39d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa498107230 0x7fa4981004c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:09.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa49d39d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49810d180 con 0x7fa498104900 2024-03-28T02:11:09.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa49d39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49810d000 con 0x7fa498101fd0 2024-03-28T02:11:09.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa49d39d640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49810d300 con 0x7fa498107230 2024-03-28T02:11:09.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 0x7fa4980fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:09.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.938+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 0x7fa4980fff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37188/0 (socket says 172.21.15.106:37188) 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4977fe640 1 -- 172.21.15.106:0/3366727810 learned_addr learned my addr 172.21.15.106:0/3366727810 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa497fff640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa498101fd0 0x7fa498101940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa49cb9c640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa498107230 0x7fa4981004c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4977fe640 1 -- 172.21.15.106:0/3366727810 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa498107230 msgr2=0x7fa4981004c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4977fe640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa498107230 0x7fa4981004c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4977fe640 1 -- 172.21.15.106:0/3366727810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa498101fd0 msgr2=0x7fa498101940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4977fe640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa498101fd0 0x7fa498101940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4977fe640 1 -- 172.21.15.106:0/3366727810 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498100d50 con 0x7fa498104900 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4977fe640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 0x7fa4980fff80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa48c000c00 tx=0x7fa48c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4957fa640 1 -- 172.21.15.106:0/3366727810 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c041070 con 0x7fa498104900 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4957fa640 1 -- 172.21.15.106:0/3366727810 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa48c004450 con 0x7fa498104900 2024-03-28T02:11:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4981a1920 con 0x7fa498104900 2024-03-28T02:11:09.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa4957fa640 1 -- 172.21.15.106:0/3366727810 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c034440 con 0x7fa498104900 2024-03-28T02:11:09.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.939+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4981a1e70 con 0x7fa498104900 2024-03-28T02:11:09.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.941+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4980fd470 con 0x7fa498104900 2024-03-28T02:11:09.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.941+0000 7fa4957fa640 1 -- 172.21.15.106:0/3366727810 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 19) v1 ==== 137308+0+0 (secure 0 0 0) 0x7fa48c044070 con 0x7fa498104900 2024-03-28T02:11:09.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.942+0000 7fa4957fa640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa4800a2840 0x7fa4800a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:09.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.942+0000 7fa497fff640 1 -- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa4800a2840 msgr2=0x7fa4800a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/3501119323 2024-03-28T02:11:09.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.942+0000 7fa4957fa640 1 -- 172.21.15.106:0/3366727810 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa48c0f0920 con 0x7fa498104900 2024-03-28T02:11:09.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.942+0000 7fa497fff640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa4800a2840 0x7fa4800a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T02:11:09.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:09.946+0000 7fa4957fa640 1 -- 172.21.15.106:0/3366727810 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa48c0bd060 con 0x7fa498104900 2024-03-28T02:11:09.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: ignoring --setuser ceph since I am not root 2024-03-28T02:11:09.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: ignoring --setgroup ceph since I am not root 2024-03-28T02:11:09.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:09 smithi153 ceph-mgr[37610]: -- 172.21.15.153:0/3050225788 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555a74fc25a0 con 0x555a74f97000 2024-03-28T02:11:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:09 smithi153 ceph-mon[36228]: pgmap v366: 129 pgs: 129 active+clean; 649 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:11:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:09 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/531924699' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T02:11:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:09 smithi153 ceph-mon[36228]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T02:11:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:09 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4221354857' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T02:11:09.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:09 smithi153 ceph-mon[36228]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T02:11:10.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.143+0000 7fa497fff640 1 -- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa4800a2840 msgr2=0x7fa4800a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/3501119323 2024-03-28T02:11:10.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.143+0000 7fa497fff640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa4800a2840 0x7fa4800a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T02:11:10.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.146+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa49802d070 con 0x7fa498104900 2024-03-28T02:11:10.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.150+0000 7fa4957fa640 1 -- 172.21.15.106:0/3366727810 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa48c0c1100 con 0x7fa498104900 2024-03-28T02:11:10.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa4800a2840 msgr2=0x7fa4800a4d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T02:11:10.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa4800a2840 0x7fa4800a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 msgr2=0x7fa4980fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:10.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 0x7fa4980fff80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa48c000c00 tx=0x7fa48c02c640 comp rx=0 tx=0).stop 2024-03-28T02:11:10.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 shutdown_connections 2024-03-28T02:11:10.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa498107230 0x7fa4981004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa498104900 0x7fa4980fff80 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7fa4800a2840 0x7fa4800a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 --2- 172.21.15.106:0/3366727810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa498101fd0 0x7fa498101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 >> 172.21.15.106:0/3366727810 conn(0x7fa4980fb980 msgr2=0x7fa498102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:10.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 shutdown_connections 2024-03-28T02:11:10.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.152+0000 7fa49d39d640 1 -- 172.21.15.106:0/3366727810 wait complete. 2024-03-28T02:11:10.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-28T02:11:10.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.269+0000 7f64fdaf7640 1 -- 172.21.15.106:0/3085365904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f81017b0 msgr2=0x7f64f8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.269+0000 7f64fdaf7640 1 --2- 172.21.15.106:0/3085365904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f81017b0 0x7f64f8101c30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009900 tx=0x7f64ec02bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.269+0000 7f64fdaf7640 1 -- 172.21.15.106:0/3085365904 shutdown_connections 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.269+0000 7f64fdaf7640 1 --2- 172.21.15.106:0/3085365904 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64f8102170 0x7f64f8108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.269+0000 7f64fdaf7640 1 --2- 172.21.15.106:0/3085365904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f81017b0 0x7f64f8101c30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.269+0000 7f64fdaf7640 1 --2- 172.21.15.106:0/3085365904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f81005c0 0x7f64f81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.269+0000 7f64fdaf7640 1 -- 172.21.15.106:0/3085365904 >> 172.21.15.106:0/3085365904 conn(0x7f64f80fbb30 msgr2=0x7f64f80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.269+0000 7f64fdaf7640 1 -- 172.21.15.106:0/3085365904 shutdown_connections 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.269+0000 7f64fdaf7640 1 -- 172.21.15.106:0/3085365904 wait complete. 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64fdaf7640 1 Processor -- start 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64fdaf7640 1 -- start start 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64fdaf7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f81005c0 0x7f64f8196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64fdaf7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f81017b0 0x7f64f8196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64fdaf7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64f8102170 0x7f64f819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64fdaf7640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f810c220 con 0x7f64f8102170 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64fdaf7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f810c0a0 con 0x7f64f81017b0 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64fdaf7640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f810c3a0 con 0x7f64f81005c0 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64f7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64f8102170 0x7f64f819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64f77fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f81005c0 0x7f64f8196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64f7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64f8102170 0x7f64f819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37210/0 (socket says 172.21.15.106:37210) 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64f7fff640 1 -- 172.21.15.106:0/1009917149 learned_addr learned my addr 172.21.15.106:0/1009917149 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64f77fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f81005c0 0x7f64f8196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:47850/0 (socket says 172.21.15.106:47850) 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.270+0000 7f64f6ffd640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f81017b0 0x7f64f8196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:10.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64f7fff640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f81005c0 msgr2=0x7f64f8196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64f7fff640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f81005c0 0x7f64f8196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64f7fff640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f81017b0 msgr2=0x7f64f8196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64f7fff640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f81017b0 0x7f64f8196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64f7fff640 1 -- 172.21.15.106:0/1009917149 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f819e480 con 0x7f64f8102170 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64f7fff640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64f8102170 0x7f64f819de10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f64f000efd0 tx=0x7f64f000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f0003b70 con 0x7f64f8102170 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f819e770 con 0x7f64f8102170 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64f0003d10 con 0x7f64f8102170 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f00109f0 con 0x7f64f8102170 2024-03-28T02:11:10.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.271+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f819ec40 con 0x7f64f8102170 2024-03-28T02:11:10.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.272+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f81078e0 con 0x7f64f8102170 2024-03-28T02:11:10.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.273+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 19) v1 ==== 137308+0+0 (secure 0 0 0) 0x7f64f0010b90 con 0x7f64f8102170 2024-03-28T02:11:10.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.274+0000 7f64f4ff9640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 0x7f64e00a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:10.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.274+0000 7f64f77fe640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 msgr2=0x7f64e00a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/3501119323 2024-03-28T02:11:10.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.274+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f64f00cf660 con 0x7f64f8102170 2024-03-28T02:11:10.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.274+0000 7f64f77fe640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 0x7f64e00a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T02:11:10.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.277+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f64f009c360 con 0x7f64f8102170 2024-03-28T02:11:10.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.475+0000 7f64f77fe640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 msgr2=0x7f64e00a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/3501119323 2024-03-28T02:11:10.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.475+0000 7f64f77fe640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 0x7f64e00a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T02:11:10.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.481+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 --> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f64f8061ae0 con 0x7f64e00a28f0 2024-03-28T02:11:10.482 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:10.144+0000 7fd131818200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T02:11:10.482 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:10.252+0000 7fd131818200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T02:11:10.491 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:10.094+0000 7f11e1ad0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T02:11:10.491 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:10.193+0000 7f11e1ad0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T02:11:10.491 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:10 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:10.222+0000 7f34a1a33200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T02:11:10.492 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:10 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:10.360+0000 7f34a1a33200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T02:11:10.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:10 smithi106 ceph-mon[31965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T02:11:10.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:10 smithi106 ceph-mon[31965]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T02:11:10.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:10 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3628939964' entity='client.admin' 2024-03-28T02:11:10.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:10 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3366727810' entity='client.admin' 2024-03-28T02:11:10.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.876+0000 7f64f77fe640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 msgr2=0x7f64e00a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/3501119323 2024-03-28T02:11:10.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:10.876+0000 7f64f77fe640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 0x7f64e00a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T02:11:10.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:10 smithi178 ceph-mon[36521]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T02:11:10.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:10 smithi178 ceph-mon[36521]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T02:11:10.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:10 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3628939964' entity='client.admin' 2024-03-28T02:11:10.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:10 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3366727810' entity='client.admin' 2024-03-28T02:11:10.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:10 smithi153 ceph-mon[36228]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T02:11:10.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:10 smithi153 ceph-mon[36228]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T02:11:10.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:10 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3628939964' entity='client.admin' 2024-03-28T02:11:10.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:10 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3366727810' entity='client.admin' 2024-03-28T02:11:11.166 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:10.845+0000 7f11e1ad0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T02:11:11.166 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:10.939+0000 7f11e1ad0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T02:11:11.166 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:11.078+0000 7f11e1ad0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T02:11:11.166 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:11.173+0000 7f11e1ad0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T02:11:11.232 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:10.953+0000 7fd131818200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T02:11:11.232 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:11.048+0000 7fd131818200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T02:11:11.232 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:11.195+0000 7fd131818200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T02:11:11.517 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:11.255+0000 7f11e1ad0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T02:11:11.564 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:11.252+0000 7f34a1a33200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T02:11:11.564 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:11.386+0000 7f34a1a33200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T02:11:11.564 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:11.571+0000 7f34a1a33200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T02:11:11.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:11.677+0000 7f64f77fe640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 msgr2=0x7f64e00a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/3501119323 2024-03-28T02:11:11.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:11.677+0000 7f64f77fe640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 0x7f64e00a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T02:11:11.704 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:11.319+0000 7fd131818200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T02:11:11.704 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:11.441+0000 7fd131818200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T02:11:11.807 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:11.524+0000 7f11e1ad0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T02:11:11.808 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:11.633+0000 7f11e1ad0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T02:11:11.808 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:11.720+0000 7f11e1ad0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T02:11:11.871 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:11.666+0000 7f34a1a33200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T02:11:11.871 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:11.796+0000 7f34a1a33200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T02:11:11.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:11.711+0000 7fd131818200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T02:11:11.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:11.824+0000 7fd131818200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T02:11:11.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:11.912+0000 7fd131818200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T02:11:12.082 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:11.814+0000 7f11e1ad0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T02:11:12.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:11.989+0000 7f11e1ad0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T02:11:12.371 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:12.091+0000 7f34a1a33200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T02:11:12.372 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:12.203+0000 7f34a1a33200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T02:11:12.372 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:12.301+0000 7f34a1a33200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T02:11:12.482 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:12.010+0000 7fd131818200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T02:11:12.482 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:12.190+0000 7fd131818200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T02:11:12.871 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:12.425+0000 7f34a1a33200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T02:11:12.872 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:12.649+0000 7f34a1a33200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T02:11:13.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:13.278+0000 7f64f77fe640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 msgr2=0x7f64e00a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6800/3501119323 2024-03-28T02:11:13.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:13.278+0000 7f64f77fe640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 0x7f64e00a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T02:11:13.332 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:12.948+0000 7f11e1ad0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T02:11:13.333 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:13.077+0000 7f11e1ad0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T02:11:13.693 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:13.412+0000 7f11e1ad0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T02:11:13.693 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:13.502+0000 7f11e1ad0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T02:11:13.732 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:13.313+0000 7fd131818200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T02:11:13.732 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:13.465+0000 7fd131818200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T02:11:14.082 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:13.699+0000 7f11e1ad0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:11:14.082 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:13.886+0000 7f11e1ad0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T02:11:14.121 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:13.800+0000 7f34a1a33200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T02:11:14.121 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:13.981+0000 7f34a1a33200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T02:11:14.194 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:13.850+0000 7fd131818200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T02:11:14.194 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:13.959+0000 7fd131818200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T02:11:14.423 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:14.174+0000 7f11e1ad0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:11:14.482 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:14 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:14.201+0000 7fd131818200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:11:14.483 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:14 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:14.410+0000 7fd131818200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T02:11:14.757 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:14.404+0000 7f34a1a33200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T02:11:14.757 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:14.524+0000 7f34a1a33200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T02:11:14.832 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:14.429+0000 7f11e1ad0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T02:11:14.832 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:14.669+0000 7f11e1ad0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T02:11:14.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:14 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:14.714+0000 7fd131818200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:11:15.121 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:14.763+0000 7f34a1a33200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:11:15.121 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:14.987+0000 7f34a1a33200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T02:11:15.306 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:15 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:15.033+0000 7fd131818200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T02:11:15.621 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:15.320+0000 7f34a1a33200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T02:11:15.732 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:15 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:15.313+0000 7fd131818200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T02:11:15.832 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:15 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:15.475+0000 7f11e1ad0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T02:11:15.917 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:15.644+0000 7f34a1a33200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T02:11:16.332 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.048+0000 7f11e1ad0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T02:11:16.333 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.153+0000 7f11e1ad0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T02:11:16.371 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:15.923+0000 7f34a1a33200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T02:11:16.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:16.399+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 20) v1 ==== 136975+0+0 (secure 0 0 0) 0x7f64f00982d0 con 0x7f64f8102170 2024-03-28T02:11:16.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:16.400+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 msgr2=0x7f64e00a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T02:11:16.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:16.400+0000 7f64f4ff9640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/3501119323,v1:172.21.15.106:6801/3501119323] conn(0x7f64e00a28f0 0x7f64e00a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:16.437 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:16.162+0000 7fd131818200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T02:11:16.655 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.376+0000 7f11e1ad0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T02:11:16.656 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: Active manager daemon x restarted 2024-03-28T02:11:16.656 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: Activating manager daemon x 2024-03-28T02:11:16.656 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T02:11:16.656 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: mgrmap e20: x(active, starting, since 0.0111187s), standbys: z, y 2024-03-28T02:11:16.656 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:11:16.656 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:11:16.656 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:11:16.656 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T02:11:16.656 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T02:11:16.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:11:16.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:11:16.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T02:11:16.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T02:11:16.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T02:11:16.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-mon[31965]: Manager daemon x is now available 2024-03-28T02:11:16.699 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: Active manager daemon x restarted 2024-03-28T02:11:16.699 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: Activating manager daemon x 2024-03-28T02:11:16.699 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: mgrmap e20: x(active, starting, since 0.0111187s), standbys: z, y 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T02:11:16.700 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:11:16.701 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:11:16.702 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T02:11:16.702 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T02:11:16.702 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T02:11:16.702 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-mon[36521]: Manager daemon x is now available 2024-03-28T02:11:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: Active manager daemon x restarted 2024-03-28T02:11:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: Activating manager daemon x 2024-03-28T02:11:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T02:11:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: mgrmap e20: x(active, starting, since 0.0111187s), standbys: z, y 2024-03-28T02:11:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T02:11:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T02:11:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T02:11:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T02:11:16.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T02:11:16.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T02:11:16.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-mon[36228]: Manager daemon x is now available 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.725+0000 7f114145f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.725+0000 7f114145f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.725+0000 7f114145f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.725+0000 7f114145f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.725+0000 7f114145f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.729+0000 7f1142c62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.729+0000 7f1142c62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.729+0000 7f1142c62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.729+0000 7f1142c62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.083 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:11:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:11:16.729+0000 7f1142c62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:11:17.121 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:16 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:16.949+0000 7f34a1a33200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T02:11:17.232 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:16.878+0000 7fd131818200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T02:11:17.232 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:16 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:16.992+0000 7fd131818200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T02:11:17.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.407+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 <== mon.0 v2:172.21.15.106:3300/0 8 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f64f00979b0 con 0x7f64f8102170 2024-03-28T02:11:17.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.407+0000 7f64f4ff9640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f64e0051c50 0x7f64e0052030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:17.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.407+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 --> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f64e003ca40 con 0x7f64e0051c50 2024-03-28T02:11:17.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.409+0000 7f64f77fe640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f64e0051c50 0x7f64e0052030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:17.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.412+0000 7f64f77fe640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f64e0051c50 0x7f64e0052030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f64f8101610 tx=0x7f64e8005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:17.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.493+0000 7f64f4ff9640 1 -- 172.21.15.106:0/1009917149 <== mgr.25315 v2:172.21.15.106:6800/2358705651 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f64e003ca40 con 0x7f64e0051c50 2024-03-28T02:11:17.485 INFO:teuthology.orchestra.run.smithi106.stdout:Schedule set for path / 2024-03-28T02:11:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.495+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f64e0051c50 msgr2=0x7f64e0052030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.495+0000 7f64fdaf7640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f64e0051c50 0x7f64e0052030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f64f8101610 tx=0x7f64e8005f90 comp rx=0 tx=0).stop 2024-03-28T02:11:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.495+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64f8102170 msgr2=0x7f64f819de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.495+0000 7f64fdaf7640 1 --2- 172.21.15.106:0/1009917149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64f8102170 0x7f64f819de10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f64f000efd0 tx=0x7f64f000c6a0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.495+0000 7f64f7fff640 1 -- 172.21.15.106:0/1009917149 reap_dead start 2024-03-28T02:11:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.495+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 shutdown_connections 2024-03-28T02:11:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.495+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 >> 172.21.15.106:0/1009917149 conn(0x7f64f80fbb30 msgr2=0x7f64f8104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.496+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 shutdown_connections 2024-03-28T02:11:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.496+0000 7f64fdaf7640 1 -- 172.21.15.106:0/1009917149 wait complete. 2024-03-28T02:11:17.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 -- 172.21.15.106:0/2773152950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae641016f0 msgr2=0x7fae64101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/2773152950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae641016f0 0x7fae64101b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fae60009860 tx=0x7fae6002bb70 comp rx=0 tx=0).stop 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 -- 172.21.15.106:0/2773152950 shutdown_connections 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/2773152950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae641032a0 0x7fae64109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/2773152950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae641028e0 0x7fae64102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/2773152950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae641016f0 0x7fae64101b10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 -- 172.21.15.106:0/2773152950 >> 172.21.15.106:0/2773152950 conn(0x7fae640fcc60 msgr2=0x7fae640ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 -- 172.21.15.106:0/2773152950 shutdown_connections 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 -- 172.21.15.106:0/2773152950 wait complete. 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 Processor -- start 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.615+0000 7fae6bf5b640 1 -- start start 2024-03-28T02:11:17.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae6bf5b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae641016f0 0x7fae641979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae6bf5b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae641028e0 0x7fae64197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae6bf5b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae641032a0 0x7fae6419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae6bf5b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6410d340 con 0x7fae641016f0 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae6bf5b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6410d1c0 con 0x7fae641028e0 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae6bf5b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6410d4c0 con 0x7fae641032a0 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae69cf7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae641016f0 0x7fae641979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae69cf7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae641016f0 0x7fae641979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:48958/0 (socket says 172.21.15.106:48958) 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae69cf7640 1 -- 172.21.15.106:0/4180671858 learned_addr learned my addr 172.21.15.106:0/4180671858 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae694f6640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae641028e0 0x7fae64197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae6a4f8640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae641032a0 0x7fae6419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae69cf7640 1 -- 172.21.15.106:0/4180671858 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae641032a0 msgr2=0x7fae6419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae69cf7640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae641032a0 0x7fae6419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae69cf7640 1 -- 172.21.15.106:0/4180671858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae641028e0 msgr2=0x7fae64197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:17.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae69cf7640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae641028e0 0x7fae64197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae69cf7640 1 -- 172.21.15.106:0/4180671858 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6419f5d0 con 0x7fae641016f0 2024-03-28T02:11:17.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae694f6640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae641028e0 0x7fae64197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:17.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae69cf7640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae641016f0 0x7fae641979b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fae6002c0b0 tx=0x7fae600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:17.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae5affd640 1 -- 172.21.15.106:0/4180671858 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60041020 con 0x7fae641016f0 2024-03-28T02:11:17.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae5affd640 1 -- 172.21.15.106:0/4180671858 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae600347b0 con 0x7fae641016f0 2024-03-28T02:11:17.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6419f860 con 0x7fae641016f0 2024-03-28T02:11:17.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.616+0000 7fae5affd640 1 -- 172.21.15.106:0/4180671858 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60039040 con 0x7fae641016f0 2024-03-28T02:11:17.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.617+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6419fd50 con 0x7fae641016f0 2024-03-28T02:11:17.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.618+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae64108a10 con 0x7fae641016f0 2024-03-28T02:11:17.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.619+0000 7fae5affd640 1 -- 172.21.15.106:0/4180671858 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fae60044020 con 0x7fae641016f0 2024-03-28T02:11:17.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.620+0000 7fae5affd640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fae500a26b0 0x7fae500a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:17.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.620+0000 7fae694f6640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fae500a26b0 0x7fae500a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:17.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.620+0000 7fae5affd640 1 -- 172.21.15.106:0/4180671858 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fae600f0450 con 0x7fae641016f0 2024-03-28T02:11:17.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.620+0000 7fae694f6640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fae500a26b0 0x7fae500a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fae64198ec0 tx=0x7fae5400a2d0 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:17.616 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.624+0000 7fae5affd640 1 -- 172.21.15.106:0/4180671858 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae6004b020 con 0x7fae641016f0 2024-03-28T02:11:17.732 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y[37606]: 2024-03-28T02:11:17.247+0000 7fd131818200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T02:11:17.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: Standby manager daemon y restarted 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: Standby manager daemon y started 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:11:17.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:17 smithi153 ceph-mon[36228]: mgrmap e21: x(active, since 1.01829s), standbys: z, y 2024-03-28T02:11:17.761 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:11:17.761 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:11:17.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T02:11:17.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T02:11:17.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: Standby manager daemon y restarted 2024-03-28T02:11:17.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: Standby manager daemon y started 2024-03-28T02:11:17.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T02:11:17.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:11:17.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T02:11:17.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:11:17.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:17 smithi106 ceph-mon[31965]: mgrmap e21: x(active, since 1.01829s), standbys: z, y 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: Standby manager daemon y restarted 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: Standby manager daemon y started 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.153:0/3861451502' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:11:17.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-mon[36521]: mgrmap e21: x(active, since 1.01829s), standbys: z, y 2024-03-28T02:11:17.806 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:17.693+0000 7f34a1a33200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T02:11:17.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.830+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 --> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7fae64061c40 con 0x7fae500a26b0 2024-03-28T02:11:17.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.886+0000 7fae5affd640 1 -- 172.21.15.106:0/4180671858 <== mgr.25315 v2:172.21.15.106:6800/2358705651 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fae64061c40 con 0x7fae500a26b0 2024-03-28T02:11:17.879 INFO:teuthology.orchestra.run.smithi106.stdout:Retention added to path / 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fae500a26b0 msgr2=0x7fae500a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fae500a26b0 0x7fae500a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fae64198ec0 tx=0x7fae5400a2d0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae641016f0 msgr2=0x7fae641979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae641016f0 0x7fae641979b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fae6002c0b0 tx=0x7fae600042f0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 shutdown_connections 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fae500a26b0 0x7fae500a4b70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae641032a0 0x7fae6419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae641028e0 0x7fae64197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 --2- 172.21.15.106:0/4180671858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae641016f0 0x7fae641979b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 >> 172.21.15.106:0/4180671858 conn(0x7fae640fcc60 msgr2=0x7fae64105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 shutdown_connections 2024-03-28T02:11:17.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:17.889+0000 7fae6bf5b640 1 -- 172.21.15.106:0/4180671858 wait complete. 2024-03-28T02:11:17.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T02:11:17.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.004+0000 7ff7e949a640 1 -- 172.21.15.106:0/4112242059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41017b0 msgr2=0x7ff7e4101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:17.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.004+0000 7ff7e949a640 1 --2- 172.21.15.106:0/4112242059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41017b0 0x7ff7e4101c30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009900 tx=0x7ff7cc02bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 -- 172.21.15.106:0/4112242059 shutdown_connections 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 --2- 172.21.15.106:0/4112242059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e4102170 0x7ff7e4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 --2- 172.21.15.106:0/4112242059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41017b0 0x7ff7e4101c30 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 --2- 172.21.15.106:0/4112242059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7e41005c0 0x7ff7e41009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 -- 172.21.15.106:0/4112242059 >> 172.21.15.106:0/4112242059 conn(0x7ff7e40fbb30 msgr2=0x7ff7e40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 -- 172.21.15.106:0/4112242059 shutdown_connections 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 -- 172.21.15.106:0/4112242059 wait complete. 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 Processor -- start 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 -- start start 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41005c0 0x7ff7e4197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e41017b0 0x7ff7e4197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:18.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7e4102170 0x7ff7e419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e410c220 con 0x7ff7e41005c0 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e410c0a0 con 0x7ff7e4102170 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e949a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e410c3a0 con 0x7ff7e41017b0 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41005c0 0x7ff7e4197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41005c0 0x7ff7e4197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:48980/0 (socket says 172.21.15.106:48980) 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e2ffd640 1 -- 172.21.15.106:0/3485373035 learned_addr learned my addr 172.21.15.106:0/3485373035 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7dbfff640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e41017b0 0x7ff7e4197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.005+0000 7ff7e37fe640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7e4102170 0x7ff7e419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e2ffd640 1 -- 172.21.15.106:0/3485373035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e41017b0 msgr2=0x7ff7e4197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e2ffd640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e41017b0 0x7ff7e4197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e2ffd640 1 -- 172.21.15.106:0/3485373035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7e4102170 msgr2=0x7ff7e419eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e2ffd640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7e4102170 0x7ff7e419eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e2ffd640 1 -- 172.21.15.106:0/3485373035 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e419f540 con 0x7ff7e41005c0 2024-03-28T02:11:18.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e2ffd640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41005c0 0x7ff7e4197920 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400e9c0 tx=0x7ff7d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:18.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e0ff9640 1 -- 172.21.15.106:0/3485373035 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d400cd80 con 0x7ff7e41005c0 2024-03-28T02:11:18.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e0ff9640 1 -- 172.21.15.106:0/3485373035 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7d40035f0 con 0x7ff7e41005c0 2024-03-28T02:11:18.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e419f830 con 0x7ff7e41005c0 2024-03-28T02:11:18.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e0ff9640 1 -- 172.21.15.106:0/3485373035 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4010690 con 0x7ff7e41005c0 2024-03-28T02:11:18.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.006+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e419fd50 con 0x7ff7e41005c0 2024-03-28T02:11:18.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.007+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e404f400 con 0x7ff7e41005c0 2024-03-28T02:11:18.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.008+0000 7ff7e0ff9640 1 -- 172.21.15.106:0/3485373035 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7ff7d40037a0 con 0x7ff7e41005c0 2024-03-28T02:11:18.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.008+0000 7ff7e0ff9640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff7d00a26b0 0x7ff7d00a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:18.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.009+0000 7ff7e0ff9640 1 -- 172.21.15.106:0/3485373035 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7ff7d40cef00 con 0x7ff7e41005c0 2024-03-28T02:11:18.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.009+0000 7ff7dbfff640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff7d00a26b0 0x7ff7d00a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:18.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.009+0000 7ff7dbfff640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff7d00a26b0 0x7ff7d00a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc02c7b0 tx=0x7ff7cc005fd0 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:18.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.013+0000 7ff7e0ff9640 1 -- 172.21.15.106:0/3485373035 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff7d409bdf0 con 0x7ff7e41005c0 2024-03-28T02:11:18.108 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:17.817+0000 7f34a1a33200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T02:11:18.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.211+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 --> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7e40fdc80 con 0x7ff7d00a26b0 2024-03-28T02:11:18.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.218+0000 7ff7e0ff9640 1 -- 172.21.15.106:0/3485373035 <== mgr.25315 v2:172.21.15.106:6800/2358705651 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7ff7e40fdc80 con 0x7ff7d00a26b0 2024-03-28T02:11:18.210 INFO:teuthology.orchestra.run.smithi106.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T02:11:17", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-28T02:11:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff7d00a26b0 msgr2=0x7ff7d00a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff7d00a26b0 0x7ff7d00a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc02c7b0 tx=0x7ff7cc005fd0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41005c0 msgr2=0x7ff7e4197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41005c0 0x7ff7e4197920 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400e9c0 tx=0x7ff7d400ee80 comp rx=0 tx=0).stop 2024-03-28T02:11:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 shutdown_connections 2024-03-28T02:11:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff7d00a26b0 0x7ff7d00a4b70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7e4102170 0x7ff7e419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e41017b0 0x7ff7e4197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 --2- 172.21.15.106:0/3485373035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7e41005c0 0x7ff7e4197920 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 >> 172.21.15.106:0/3485373035 conn(0x7ff7e40fbb30 msgr2=0x7ff7e40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 shutdown_connections 2024-03-28T02:11:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.220+0000 7ff7e949a640 1 -- 172.21.15.106:0/3485373035 wait complete. 2024-03-28T02:11:18.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T02:11:18.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.346+0000 7f6de190c640 1 -- 172.21.15.106:0/1969317914 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc1016d0 msgr2=0x7f6ddc101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:18.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.346+0000 7f6de190c640 1 --2- 172.21.15.106:0/1969317914 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc1016d0 0x7f6ddc101af0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009900 tx=0x7f6dcc02bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.346+0000 7f6de190c640 1 -- 172.21.15.106:0/1969317914 shutdown_connections 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.346+0000 7f6de190c640 1 --2- 172.21.15.106:0/1969317914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ddc103280 0x7f6ddc109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.346+0000 7f6de190c640 1 --2- 172.21.15.106:0/1969317914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc1028c0 0x7f6ddc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.346+0000 7f6de190c640 1 --2- 172.21.15.106:0/1969317914 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc1016d0 0x7f6ddc101af0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.346+0000 7f6de190c640 1 -- 172.21.15.106:0/1969317914 >> 172.21.15.106:0/1969317914 conn(0x7f6ddc0fcc60 msgr2=0x7f6ddc0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.346+0000 7f6de190c640 1 -- 172.21.15.106:0/1969317914 shutdown_connections 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.346+0000 7f6de190c640 1 -- 172.21.15.106:0/1969317914 wait complete. 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6de190c640 1 Processor -- start 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6de190c640 1 -- start start 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6de190c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc1016d0 0x7f6ddc075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6de190c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ddc1028c0 0x7f6ddc073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6de190c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc103280 0x7f6ddc073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6de190c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc10d350 con 0x7f6ddc103280 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6de190c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc10d1d0 con 0x7f6ddc1016d0 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6de190c640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc10d4d0 con 0x7f6ddc1028c0 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc103280 0x7f6ddc073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc1016d0 0x7f6ddc075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc1016d0 0x7f6ddc075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:40808/0 (socket says 172.21.15.106:40808) 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc103280 0x7f6ddc073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:48998/0 (socket says 172.21.15.106:48998) 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddaffd640 1 -- 172.21.15.106:0/3389461295 learned_addr learned my addr 172.21.15.106:0/3389461295 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6dda7fc640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ddc1028c0 0x7f6ddc073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddaffd640 1 -- 172.21.15.106:0/3389461295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ddc1028c0 msgr2=0x7f6ddc073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddaffd640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ddc1028c0 0x7f6ddc073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddaffd640 1 -- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc103280 msgr2=0x7f6ddc073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddaffd640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc103280 0x7f6ddc073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddaffd640 1 -- 172.21.15.106:0/3389461295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc074860 con 0x7f6ddc1016d0 2024-03-28T02:11:18.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6dda7fc640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ddc1028c0 0x7f6ddc073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:18.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.347+0000 7f6ddb7fe640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc103280 0x7f6ddc073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:18.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.348+0000 7f6ddaffd640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc1016d0 0x7f6ddc075450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc000c00 tx=0x7f6dcc02c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:18.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.348+0000 7f6de090a640 1 -- 172.21.15.106:0/3389461295 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc041070 con 0x7f6ddc1016d0 2024-03-28T02:11:18.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.348+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc1a19d0 con 0x7f6ddc1016d0 2024-03-28T02:11:18.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.348+0000 7f6de090a640 1 -- 172.21.15.106:0/3389461295 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6dcc004450 con 0x7f6ddc1016d0 2024-03-28T02:11:18.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.348+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc1a1f70 con 0x7f6ddc1016d0 2024-03-28T02:11:18.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.348+0000 7f6de090a640 1 -- 172.21.15.106:0/3389461295 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc04b060 con 0x7f6ddc1016d0 2024-03-28T02:11:18.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.349+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ddc108990 con 0x7f6ddc1016d0 2024-03-28T02:11:18.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.350+0000 7f6de090a640 1 -- 172.21.15.106:0/3389461295 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f6dcc044070 con 0x7f6ddc1016d0 2024-03-28T02:11:18.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.351+0000 7f6de090a640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f6dc40a26b0 0x7f6dc40a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:18.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.351+0000 7f6de090a640 1 -- 172.21.15.106:0/3389461295 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f6dcc0f0b10 con 0x7f6ddc1016d0 2024-03-28T02:11:18.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.351+0000 7f6dda7fc640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f6dc40a26b0 0x7f6dc40a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:18.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.351+0000 7f6dda7fc640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f6dc40a26b0 0x7f6dc40a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0751c0 tx=0x7f6dd4009500 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:18.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.355+0000 7f6de090a640 1 -- 172.21.15.106:0/3389461295 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6dcc0bd440 con 0x7f6ddc1016d0 2024-03-28T02:11:18.371 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z[38384]: 2024-03-28T02:11:18.114+0000 7f34a1a33200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T02:11:18.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.463+0000 7f6de090a640 1 -- 172.21.15.106:0/3389461295 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 22) v1 ==== 137411+0+0 (secure 0 0 0) 0x7f6dcc0b9ab0 con 0x7f6ddc1016d0 2024-03-28T02:11:18.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.586+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 --> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6ddc061ae0 con 0x7f6dc40a26b0 2024-03-28T02:11:18.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: [28/Mar/2024:02:11:17] ENGINE Bus STARTING 2024-03-28T02:11:18.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:18.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:11:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: [28/Mar/2024:02:11:17] ENGINE Serving on http://172.21.15.106:8765 2024-03-28T02:11:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: Standby manager daemon z restarted 2024-03-28T02:11:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: Standby manager daemon z started 2024-03-28T02:11:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T02:11:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:11:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T02:11:18.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:18 smithi106 ceph-mon[31965]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:11:18.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.592+0000 7f6de090a640 1 -- 172.21.15.106:0/3389461295 <== mgr.25315 v2:172.21.15.106:6800/2358705651 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f6ddc061ae0 con 0x7f6dc40a26b0 2024-03-28T02:11:18.584 INFO:teuthology.orchestra.run.smithi106.stdout:/ 1m 6m3h 2024-03-28T02:11:18.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f6dc40a26b0 msgr2=0x7f6dc40a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f6dc40a26b0 0x7f6dc40a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0751c0 tx=0x7f6dd4009500 comp rx=0 tx=0).stop 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc1016d0 msgr2=0x7f6ddc075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc1016d0 0x7f6ddc075450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc000c00 tx=0x7f6dcc02c640 comp rx=0 tx=0).stop 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 shutdown_connections 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f6dc40a26b0 0x7f6dc40a4b70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc103280 0x7f6ddc073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ddc1028c0 0x7f6ddc073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 --2- 172.21.15.106:0/3389461295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc1016d0 0x7f6ddc075450 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 >> 172.21.15.106:0/3389461295 conn(0x7f6ddc0fcc60 msgr2=0x7f6ddc105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 shutdown_connections 2024-03-28T02:11:18.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:11:18.594+0000 7f6de190c640 1 -- 172.21.15.106:0/3389461295 wait complete. 2024-03-28T02:11:18.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-28T02:11:18.634 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-28T02:11:18.645 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-28T02:11:18.691 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:11:18.691 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-28T02:11:18.693 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T02:11:18.727 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T02:11:18.728 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T02:11:18.728 INFO:tasks.workunit:timeout=3h 2024-03-28T02:11:18.728 INFO:tasks.workunit:cleanup=True 2024-03-28T02:11:18.728 DEBUG:teuthology.orchestra.run.smithi106:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T02:11:18.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: [28/Mar/2024:02:11:17] ENGINE Bus STARTING 2024-03-28T02:11:18.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:18.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:11:18.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: [28/Mar/2024:02:11:17] ENGINE Serving on http://172.21.15.106:8765 2024-03-28T02:11:18.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: Standby manager daemon z restarted 2024-03-28T02:11:18.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: Standby manager daemon z started 2024-03-28T02:11:18.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T02:11:18.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:11:18.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T02:11:18.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:18 smithi153 ceph-mon[36228]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:11:18.743 INFO:teuthology.orchestra.run.smithi106.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T02:11:18.743 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T02:11:18.743 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-28T02:11:18.743 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T02:11:18.743 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T02:11:18.743 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-28 02:09:10.657512132 +0000 2024-03-28T02:11:18.743 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-28 02:09:10.657512132 +0000 2024-03-28T02:11:18.743 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-28 02:11:07.090306504 +0000 2024-03-28T02:11:18.743 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-28 02:09:10.657512132 +0000 2024-03-28T02:11:18.744 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T02:11:18.744 DEBUG:teuthology.orchestra.run.smithi106:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T02:11:18.820 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T02:11:18.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: [28/Mar/2024:02:11:17] ENGINE Bus STARTING 2024-03-28T02:11:18.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:11:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: [28/Mar/2024:02:11:17] ENGINE Serving on http://172.21.15.106:8765 2024-03-28T02:11:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: Standby manager daemon z restarted 2024-03-28T02:11:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: Standby manager daemon z started 2024-03-28T02:11:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T02:11:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T02:11:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T02:11:18.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:18 smithi178 ceph-mon[36521]: from='mgr.? 172.21.15.178:0/1246224991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T02:11:18.875 INFO:tasks.workunit.client.0.smithi106.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T02:11:19.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: [28/Mar/2024:02:11:17] ENGINE Serving on https://172.21.15.106:7150 2024-03-28T02:11:19.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: [28/Mar/2024:02:11:17] ENGINE Bus STARTED 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:19 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: [28/Mar/2024:02:11:17] ENGINE Serving on https://172.21.15.106:7150 2024-03-28T02:11:19.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: [28/Mar/2024:02:11:17] ENGINE Bus STARTED 2024-03-28T02:11:19.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:19.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:19.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:11:19.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-03-28T02:11:19.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:19 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: [28/Mar/2024:02:11:17] ENGINE Serving on https://172.21.15.106:7150 2024-03-28T02:11:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: [28/Mar/2024:02:11:17] ENGINE Bus STARTED 2024-03-28T02:11:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-03-28T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:19 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:11:20.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:20 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:11:20.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:20 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:20.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:11:20.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:11:20.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:20 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:11:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi106 to 176.2M 2024-03-28T02:11:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Unable to set osd_memory_target on smithi106 to 184782438: error parsing value: Value '184782438' is below minimum 939524096 2024-03-28T02:11:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-03-28T02:11:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Unable to set osd_memory_target on smithi153 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-28T02:11:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi178 to 174.1M 2024-03-28T02:11:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Unable to set osd_memory_target on smithi178 to 182604083: error parsing value: Value '182604083' is below minimum 939524096 2024-03-28T02:11:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-28T02:11:21.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:11:21.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:11:21.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:11:21.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi106 to 176.2M 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Unable to set osd_memory_target on smithi106 to 184782438: error parsing value: Value '184782438' is below minimum 939524096 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Unable to set osd_memory_target on smithi153 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi178 to 174.1M 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Unable to set osd_memory_target on smithi178 to 182604083: error parsing value: Value '182604083' is below minimum 939524096 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:11:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:11:21.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:11:21.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:11:21.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi106 to 176.2M 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Unable to set osd_memory_target on smithi106 to 184782438: error parsing value: Value '184782438' is below minimum 939524096 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Unable to set osd_memory_target on smithi153 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi178 to 174.1M 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Unable to set osd_memory_target on smithi178 to 182604083: error parsing value: Value '182604083' is below minimum 939524096 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-28T02:11:21.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:21.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:11:21.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:11:21.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:11:21.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:11:22.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:22 smithi153 ceph-mon[36228]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:11:22.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:22 smithi153 ceph-mon[36228]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:11:22.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:22 smithi153 ceph-mon[36228]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:11:22.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:22 smithi106 ceph-mon[31965]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:11:22.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:22 smithi106 ceph-mon[31965]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:11:22.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:22 smithi106 ceph-mon[31965]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:11:22.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:22 smithi178 ceph-mon[36521]: Updating smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:11:22.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:22 smithi178 ceph-mon[36521]: Updating smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:11:22.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:22 smithi178 ceph-mon[36521]: Updating smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T02:11:23.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:23 smithi153 ceph-mon[36228]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 0 B/s wr, 25 op/s 2024-03-28T02:11:23.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:23 smithi106 ceph-mon[31965]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 0 B/s wr, 25 op/s 2024-03-28T02:11:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:23 smithi178 ceph-mon[36521]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 0 B/s wr, 25 op/s 2024-03-28T02:11:25.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:25 smithi153 ceph-mon[36228]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T02:11:25.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:25 smithi106 ceph-mon[31965]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T02:11:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:25 smithi178 ceph-mon[36521]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T02:11:27.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:27 smithi153 ceph-mon[36228]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T02:11:27.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:27 smithi106 ceph-mon[31965]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T02:11:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:27 smithi178 ceph-mon[36521]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T02:11:29.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:29 smithi153 ceph-mon[36228]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-28T02:11:29.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:29 smithi106 ceph-mon[31965]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-28T02:11:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:29 smithi178 ceph-mon[36521]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-28T02:11:31.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:31 smithi153 ceph-mon[36228]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-28T02:11:31.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:31 smithi106 ceph-mon[31965]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-28T02:11:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 -- 172.21.15.106:0/2097420584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920107720 msgr2=0x7f7920107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 --2- 172.21.15.106:0/2097420584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920107720 0x7f7920107b00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7910009930 tx=0x7f791002bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 -- 172.21.15.106:0/2097420584 shutdown_connections 2024-03-28T02:11:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 --2- 172.21.15.106:0/2097420584 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f792006a860 0x7f792010e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 --2- 172.21.15.106:0/2097420584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7920069ea0 0x7f792006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 --2- 172.21.15.106:0/2097420584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920107720 0x7f7920107b00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 -- 172.21.15.106:0/2097420584 >> 172.21.15.106:0/2097420584 conn(0x7f7920077370 msgr2=0x7f7920077780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 -- 172.21.15.106:0/2097420584 shutdown_connections 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 -- 172.21.15.106:0/2097420584 wait complete. 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 Processor -- start 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.840+0000 7f7926b76640 1 -- start start 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7926b76640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920069ea0 0x7f792019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7926b76640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f792006a860 0x7f792019a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7926b76640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7920107720 0x7f792019e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7926b76640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7920111a90 con 0x7f7920069ea0 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7926b76640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7920111910 con 0x7f7920107720 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7926b76640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7920111c10 con 0x7f792006a860 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7924912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920069ea0 0x7f792019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7924912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920069ea0 0x7f792019a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51088/0 (socket says 172.21.15.106:51088) 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7924912640 1 -- 172.21.15.106:0/1279388160 learned_addr learned my addr 172.21.15.106:0/1279388160 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f791ffff640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f792006a860 0x7f792019a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7925113640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7920107720 0x7f792019e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7924912640 1 -- 172.21.15.106:0/1279388160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f792006a860 msgr2=0x7f792019a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7924912640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f792006a860 0x7f792019a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7924912640 1 -- 172.21.15.106:0/1279388160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7920107720 msgr2=0x7f792019e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7924912640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7920107720 0x7f792019e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7924912640 1 -- 172.21.15.106:0/1279388160 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792019f000 con 0x7f7920069ea0 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7925113640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7920107720 0x7f792019e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7924912640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920069ea0 0x7f792019a010 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7910009900 tx=0x7f7910004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f791dffb640 1 -- 172.21.15.106:0/1279388160 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910041070 con 0x7f7920069ea0 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f791dffb640 1 -- 172.21.15.106:0/1279388160 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79100054e0 con 0x7f7920069ea0 2024-03-28T02:11:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f7926b76640 1 -- 172.21.15.106:0/1279388160 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792019f290 con 0x7f7920069ea0 2024-03-28T02:11:31.838 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.841+0000 7f791dffb640 1 -- 172.21.15.106:0/1279388160 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910003c20 con 0x7f7920069ea0 2024-03-28T02:11:31.838 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.842+0000 7f7926b76640 1 -- 172.21.15.106:0/1279388160 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79201a6b80 con 0x7f7920069ea0 2024-03-28T02:11:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f791dffb640 1 -- 172.21.15.106:0/1279388160 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 23) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f7910044070 con 0x7f7920069ea0 2024-03-28T02:11:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7929835740 1 Processor -- start 2024-03-28T02:11:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7929835740 1 -- start start 2024-03-28T02:11:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f791dffb640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f79080a2b30 0x7f79080a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7929835740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e16eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7929835740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e1778c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7929835740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 0x55dd6e17a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7929835740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6e0d2b10 con 0x55dd6e16e710 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7929835740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6e07b900 con 0x55dd6e16f030 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7929835740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6e0cebb0 con 0x55dd6e177e00 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7924912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e16eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.844+0000 7f7925113640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f79080a2b30 0x7f79080a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f7924912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e16eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51102/0 (socket says 172.21.15.106:51102) 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f791ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e1778c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f7924912640 1 -- 172.21.15.106:0/1170819513 learned_addr learned my addr 172.21.15.106:0/1170819513 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f791dffb640 1 -- 172.21.15.106:0/1279388160 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f79100f0e20 con 0x7f7920069ea0 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f7925113640 1 --2- 172.21.15.106:0/1170819513 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 0x55dd6e17a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f791ffff640 1 -- 172.21.15.106:0/1170819513 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 msgr2=0x55dd6e17a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f791ffff640 1 --2- 172.21.15.106:0/1170819513 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 0x55dd6e17a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f791ffff640 1 -- 172.21.15.106:0/1170819513 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 msgr2=0x55dd6e16eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f791ffff640 1 --2- 172.21.15.106:0/1170819513 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e16eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f791ffff640 1 -- 172.21.15.106:0/1170819513 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd6dde7210 con 0x55dd6e16f030 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f7924912640 1 --2- 172.21.15.106:0/1170819513 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e16eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f7925113640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f79080a2b30 0x7f79080a4ff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7918005c40 tx=0x7f7918009e50 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f791ffff640 1 --2- 172.21.15.106:0/1170819513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e1778c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7914009bf0 tx=0x7f7914011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7cef95eab72ce42 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f7906ffd640 1 -- 172.21.15.106:0/1170819513 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791401e070 con 0x55dd6e16f030 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f7906ffd640 1 -- 172.21.15.106:0/1170819513 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7914009da0 con 0x55dd6e16f030 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.845+0000 7f7906ffd640 1 -- 172.21.15.106:0/1170819513 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791400e870 con 0x55dd6e16f030 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 -- 172.21.15.106:0/1170819513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 msgr2=0x55dd6e1778c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 --2- 172.21.15.106:0/1170819513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e1778c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7914009bf0 tx=0x7f7914011810 comp rx=0 tx=0).stop 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 -- 172.21.15.106:0/1170819513 shutdown_connections 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 --2- 172.21.15.106:0/1170819513 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 0x55dd6e17a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 --2- 172.21.15.106:0/1170819513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e1778c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 --2- 172.21.15.106:0/1170819513 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e16eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 -- 172.21.15.106:0/1170819513 >> 172.21.15.106:0/1170819513 conn(0x55dd6e169030 msgr2=0x55dd6e169410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 -- 172.21.15.106:0/1170819513 shutdown_connections 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 -- 172.21.15.106:0/1170819513 wait complete. 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 -- start start 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.846+0000 7f7929835740 1 Processor -- start 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f7929835740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 0x55dd6e174dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f7929835740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e175310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f7929835740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 0x55dd6e175850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f7929835740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6e17aa30 con 0x55dd6e16f030 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f7929835740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6e08ed50 con 0x55dd6e177e00 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f7929835740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6e17abb0 con 0x55dd6e16e710 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f7929835740 10 client.0 _ll_register_callbacks cb 0x55dd6e08e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f791ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e175310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.847 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f791ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e175310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51112/0 (socket says 172.21.15.106:51112) 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f791ffff640 1 -- 172.21.15.106:0/1758713871 learned_addr learned my addr 172.21.15.106:0/1758713871 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f7924912640 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 0x55dd6e174dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f7925113640 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 0x55dd6e175850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f791ffff640 1 -- 172.21.15.106:0/1758713871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 msgr2=0x55dd6e174dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f791ffff640 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 0x55dd6e174dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f791ffff640 1 -- 172.21.15.106:0/1758713871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 msgr2=0x55dd6e175850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f791ffff640 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 0x55dd6e175850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f791ffff640 1 -- 172.21.15.106:0/1758713871 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd6deb3620 con 0x55dd6e16f030 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f791ffff640 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e175310 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7914011d10 tx=0x7f791400e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c607a7af2b4cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f78fffff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.847+0000 7f78fffff640 1 -- 172.21.15.106:0/1758713871 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791401e040 con 0x55dd6e16f030 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f78fffff640 1 -- 172.21.15.106:0/1758713871 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f7914005040 con 0x55dd6e16f030 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7929835740 1 -- 172.21.15.106:0/1758713871 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7914009da0 con 0x55dd6e16f030 2024-03-28T02:11:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f78fffff640 1 -- 172.21.15.106:0/1758713871 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7914032050 con 0x55dd6e16f030 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7929835740 10 client.15636 fetch_fsmap learned FSMap version 29 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7929835740 1 -- 172.21.15.106:0/1758713871 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55dd6e186d50 con 0x55dd6e16f030 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f78fffff640 1 -- 172.21.15.106:0/1758713871 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7914009da0 con 0x55dd6e16f030 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f78fffff640 1 -- 172.21.15.106:0/1758713871 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f7914019450 con 0x55dd6e16f030 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7929835740 10 client.15636 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7929835740 10 client.15636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7929835740 20 client.15636 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7929835740 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55dd6e188150 0x55dd6e18a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7929835740 4 client.15636 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7929835740 1 -- 172.21.15.106:0/1758713871 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd6e0e0050 con 0x55dd6e188150 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.848+0000 7f7924912640 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55dd6e188150 0x55dd6e18a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7924912640 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55dd6e188150 0x55dd6e18a540 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a18042bb513bade7 server_cookie=7cb9b7809697d679 in_seq=0 out_seq=0 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f78fffff640 10 client.15636 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f78fffff640 1 -- 172.21.15.106:0/1758713871 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dd6e0e0050 con 0x55dd6e188150 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f78fffff640 10 client.15636 handle_command_reply: tid=0 2024-03-28T02:11:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7929835740 1 client.15636 shutdown 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7929835740 1 -- 172.21.15.106:0/1758713871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 msgr2=0x55dd6e175310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7929835740 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e175310 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7914011d10 tx=0x7f791400e4d0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7929835740 1 -- 172.21.15.106:0/1758713871 shutdown_connections 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7929835740 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55dd6e188150 0x55dd6e18a540 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7929835740 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 0x55dd6e175850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7929835740 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e175310 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7929835740 1 --2- 172.21.15.106:0/1758713871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 0x55dd6e174dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f78fffff640 0 client.15636 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.849+0000 7f7929835740 1 -- 172.21.15.106:0/1758713871 >> 172.21.15.106:0/1758713871 conn(0x55dd6e169030 msgr2=0x55dd6e16ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.850+0000 7f7929835740 1 -- 172.21.15.106:0/1758713871 shutdown_connections 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.850+0000 7f7929835740 1 -- 172.21.15.106:0/1758713871 wait complete. 2024-03-28T02:11:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.850+0000 7f7929835740 20 client.15636 trim_cache size 0 max 16384 2024-03-28T02:11:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.863+0000 7f7929835740 1 Processor -- start 2024-03-28T02:11:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.863+0000 7f7929835740 1 -- start start 2024-03-28T02:11:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.863+0000 7f7929835740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 0x55dd6e16ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.863+0000 7f7929835740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e1832f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.863+0000 7f7929835740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 0x55dd6e183830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.863+0000 7f7929835740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79201b0980 con 0x55dd6e16f030 2024-03-28T02:11:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.863+0000 7f7929835740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79201b2300 con 0x55dd6e177e00 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.863+0000 7f7929835740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79201b0fe0 con 0x55dd6e16e710 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f791ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e1832f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f791ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e1832f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51126/0 (socket says 172.21.15.106:51126) 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f7924912640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 0x55dd6e16ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f791ffff640 1 -- 172.21.15.106:0/2412611517 learned_addr learned my addr 172.21.15.106:0/2412611517 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f7925113640 1 --2- 172.21.15.106:0/2412611517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 0x55dd6e183830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f791ffff640 1 -- 172.21.15.106:0/2412611517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 msgr2=0x55dd6e16ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f791ffff640 1 --2- 172.21.15.106:0/2412611517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 0x55dd6e16ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f791ffff640 1 -- 172.21.15.106:0/2412611517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 msgr2=0x55dd6e183830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f791ffff640 1 --2- 172.21.15.106:0/2412611517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 0x55dd6e183830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f791ffff640 1 -- 172.21.15.106:0/2412611517 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd6e17a910 con 0x55dd6e16f030 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f791ffff640 1 --2- 172.21.15.106:0/2412611517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e1832f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7914009b40 tx=0x7f7914005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef2adffe1310fde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f79067fc640 1 -- 172.21.15.106:0/2412611517 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791401e070 con 0x55dd6e16f030 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f79067fc640 1 -- 172.21.15.106:0/2412611517 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7914029070 con 0x55dd6e16f030 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.864+0000 7f79067fc640 1 -- 172.21.15.106:0/2412611517 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79140048c0 con 0x55dd6e16f030 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 -- 172.21.15.106:0/2412611517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 msgr2=0x55dd6e1832f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 --2- 172.21.15.106:0/2412611517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e1832f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7914009b40 tx=0x7f7914005f90 comp rx=0 tx=0).stop 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 -- 172.21.15.106:0/2412611517 shutdown_connections 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 --2- 172.21.15.106:0/2412611517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e177e00 0x55dd6e183830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 --2- 172.21.15.106:0/2412611517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16f030 0x55dd6e1832f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 --2- 172.21.15.106:0/2412611517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e16e710 0x55dd6e16ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 -- 172.21.15.106:0/2412611517 >> 172.21.15.106:0/2412611517 conn(0x55dd6e169030 msgr2=0x55dd6e16a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 -- 172.21.15.106:0/2412611517 shutdown_connections 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 -- 172.21.15.106:0/2412611517 wait complete. 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 -- start start 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.865+0000 7f7929835740 1 Processor -- start 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7929835740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e17b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7929835740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e17ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7929835740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 0x55dd6e17bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7929835740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79201b1ca0 con 0x55dd6e16e710 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7929835740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79201b1640 con 0x55dd6e16f030 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7929835740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79201aeb40 con 0x55dd6e177e00 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7929835740 10 client.0 _ll_register_callbacks cb 0x7f791401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7924912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e17b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7924912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e17b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51128/0 (socket says 172.21.15.106:51128) 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7924912640 1 -- 172.21.15.106:0/2792057420 learned_addr learned my addr 172.21.15.106:0/2792057420 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f791ffff640 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e17ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7925113640 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 0x55dd6e17bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7924912640 1 -- 172.21.15.106:0/2792057420 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 msgr2=0x55dd6e17bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7924912640 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 0x55dd6e17bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7924912640 1 -- 172.21.15.106:0/2792057420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 msgr2=0x55dd6e17ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7924912640 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e17ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7924912640 1 -- 172.21.15.106:0/2792057420 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd6e1e2b40 con 0x55dd6e16e710 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f791ffff640 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e17ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7924912640 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e17b4f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f79100bcda0 tx=0x7f79100036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d48af65e8c52b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f78fffff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f78fffff640 1 -- 172.21.15.106:0/2792057420 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791004b040 con 0x55dd6e16e710 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f78fffff640 1 -- 172.21.15.106:0/2792057420 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f79100dbe00 con 0x55dd6e16e710 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.866+0000 7f7929835740 1 -- 172.21.15.106:0/2792057420 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7914029070 con 0x55dd6e16e710 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f78fffff640 1 -- 172.21.15.106:0/2792057420 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7910004c80 con 0x55dd6e16e710 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f7929835740 10 client.15642 fetch_fsmap learned FSMap version 29 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f7929835740 1 -- 172.21.15.106:0/2792057420 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55dd6e1e3eb0 con 0x55dd6e16e710 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f78fffff640 1 -- 172.21.15.106:0/2792057420 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7914029070 con 0x55dd6e16e710 2024-03-28T02:11:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f78fffff640 1 -- 172.21.15.106:0/2792057420 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f79100b9710 con 0x55dd6e16e710 2024-03-28T02:11:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f7929835740 10 client.15642 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:11:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f7929835740 10 client.15642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:11:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f7929835740 20 client.15642 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f7929835740 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55dd6e1e4560 0x55dd6e1e6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f7929835740 4 client.15642 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:11:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f7929835740 1 -- 172.21.15.106:0/2792057420 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55dd6e0da3a0 con 0x55dd6e1e4560 2024-03-28T02:11:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f791ffff640 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55dd6e1e4560 0x55dd6e1e6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f78fffff640 10 client.15642 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.867+0000 7f791ffff640 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55dd6e1e4560 0x55dd6e1e6950 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93933fcb8237f63e server_cookie=ddf7ac0566a1fe43 in_seq=0 out_seq=0 2024-03-28T02:11:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:31 smithi178 ceph-mon[36521]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-28T02:11:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f78fffff640 1 -- 172.21.15.106:0/2792057420 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55dd6e0da3a0 con 0x55dd6e1e4560 2024-03-28T02:11:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f78fffff640 10 client.15642 handle_command_reply: tid=0 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 client.15642 shutdown 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 -- 172.21.15.106:0/2792057420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 msgr2=0x55dd6e17b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e17b4f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f79100bcda0 tx=0x7f79100036f0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 -- 172.21.15.106:0/2792057420 shutdown_connections 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55dd6e1e4560 0x55dd6e1e6950 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55dd6e177e00 0x55dd6e17bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd6e16f030 0x55dd6e17ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 --2- 172.21.15.106:0/2792057420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dd6e16e710 0x55dd6e17b4f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f78fffff640 0 client.15642 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 -- 172.21.15.106:0/2792057420 >> 172.21.15.106:0/2792057420 conn(0x55dd6e169030 msgr2=0x55dd6e1732a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 -- 172.21.15.106:0/2792057420 shutdown_connections 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 1 -- 172.21.15.106:0/2792057420 wait complete. 2024-03-28T02:11:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.879+0000 7f7929835740 20 client.15642 trim_cache size 0 max 16384 2024-03-28T02:11:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:11:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:11:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:11:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 -- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f79080a2b30 msgr2=0x7f79080a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f79080a2b30 0x7f79080a4ff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7918005c40 tx=0x7f7918009e50 comp rx=0 tx=0).stop 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 -- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920069ea0 msgr2=0x7f792019a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920069ea0 0x7f792019a010 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7910009900 tx=0x7f7910004270 comp rx=0 tx=0).stop 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 -- 172.21.15.106:0/1279388160 shutdown_connections 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f79080a2b30 0x7f79080a4ff0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7920107720 0x7f792019e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f792006a860 0x7f792019a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 --2- 172.21.15.106:0/1279388160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7920069ea0 0x7f792019a010 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 -- 172.21.15.106:0/1279388160 >> 172.21.15.106:0/1279388160 conn(0x7f7920077370 msgr2=0x7f79200fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 -- 172.21.15.106:0/1279388160 shutdown_connections 2024-03-28T02:11:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:31.880+0000 7f7926b76640 1 -- 172.21.15.106:0/1279388160 wait complete. 2024-03-28T02:11:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:33 smithi153 ceph-mon[36228]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 78 KiB/s wr, 55 op/s 2024-03-28T02:11:33.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:33 smithi106 ceph-mon[31965]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 78 KiB/s wr, 55 op/s 2024-03-28T02:11:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:33 smithi178 ceph-mon[36521]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 78 KiB/s wr, 55 op/s 2024-03-28T02:11:35.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:35 smithi153 ceph-mon[36228]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 79 KiB/s wr, 49 op/s 2024-03-28T02:11:35.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:35 smithi106 ceph-mon[31965]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 79 KiB/s wr, 49 op/s 2024-03-28T02:11:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:35 smithi178 ceph-mon[36521]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 79 KiB/s wr, 49 op/s 2024-03-28T02:11:37.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:37 smithi153 ceph-mon[36228]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T02:11:37.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:37 smithi106 ceph-mon[31965]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T02:11:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:37 smithi178 ceph-mon[36521]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T02:11:39.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:39 smithi153 ceph-mon[36228]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-03-28T02:11:39.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:39 smithi106 ceph-mon[31965]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-03-28T02:11:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:39 smithi178 ceph-mon[36521]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-03-28T02:11:41.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:41 smithi153 ceph-mon[36228]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T02:11:41.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:41 smithi106 ceph-mon[31965]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T02:11:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:41 smithi178 ceph-mon[36521]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T02:11:43.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:43 smithi153 ceph-mon[36228]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-03-28T02:11:43.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:43 smithi106 ceph-mon[31965]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-03-28T02:11:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:43 smithi178 ceph-mon[36521]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-03-28T02:11:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.973+0000 7f85c3234640 1 -- 172.21.15.106:0/1172592262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 msgr2=0x7f85bc10e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.973+0000 7f85c3234640 1 --2- 172.21.15.106:0/1172592262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 0x7f85bc10e520 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009cd0 tx=0x7f85b402c0f0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.973+0000 7f85c3234640 1 -- 172.21.15.106:0/1172592262 shutdown_connections 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.973+0000 7f85c3234640 1 --2- 172.21.15.106:0/1172592262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 0x7f85bc10e520 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.973+0000 7f85c3234640 1 --2- 172.21.15.106:0/1172592262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85bc0ffb40 0x7f85bc0fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.973+0000 7f85c3234640 1 --2- 172.21.15.106:0/1172592262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc107660 0x7f85bc107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.973+0000 7f85c3234640 1 -- 172.21.15.106:0/1172592262 >> 172.21.15.106:0/1172592262 conn(0x7f85bc076ac0 msgr2=0x7f85bc0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.973+0000 7f85c3234640 1 -- 172.21.15.106:0/1172592262 shutdown_connections 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.973+0000 7f85c3234640 1 -- 172.21.15.106:0/1172592262 wait complete. 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c3234640 1 Processor -- start 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c3234640 1 -- start start 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c3234640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc0ffb40 0x7f85bc197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c3234640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 0x7f85bc198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c3234640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85bc107660 0x7f85bc19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c3234640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc111a70 con 0x7f85bc100500 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c3234640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1118f0 con 0x7f85bc0ffb40 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c3234640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc111bf0 con 0x7f85bc107660 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 0x7f85bc198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 0x7f85bc198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50036/0 (socket says 172.21.15.106:50036) 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c0fd0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc0ffb40 0x7f85bc197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85bbfff640 1 -- 172.21.15.106:0/1240999043 learned_addr learned my addr 172.21.15.106:0/1240999043 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85c17d1640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85bc107660 0x7f85bc19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85bbfff640 1 -- 172.21.15.106:0/1240999043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85bc107660 msgr2=0x7f85bc19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85bbfff640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85bc107660 0x7f85bc19c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85bbfff640 1 -- 172.21.15.106:0/1240999043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc0ffb40 msgr2=0x7f85bc197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85bbfff640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc0ffb40 0x7f85bc197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.974+0000 7f85bbfff640 1 -- 172.21.15.106:0/1240999043 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc19cee0 con 0x7f85bc100500 2024-03-28T02:11:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.975+0000 7f85bbfff640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 0x7f85bc198430 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f85a800e9f0 tx=0x7f85a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.975+0000 7f85b9ffb640 1 -- 172.21.15.106:0/1240999043 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a800cd80 con 0x7f85bc100500 2024-03-28T02:11:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.975+0000 7f85c3234640 1 -- 172.21.15.106:0/1240999043 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc19d1d0 con 0x7f85bc100500 2024-03-28T02:11:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.975+0000 7f85b9ffb640 1 -- 172.21.15.106:0/1240999043 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85a8005500 con 0x7f85bc100500 2024-03-28T02:11:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.975+0000 7f85b9ffb640 1 -- 172.21.15.106:0/1240999043 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8010690 con 0x7f85bc100500 2024-03-28T02:11:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.975+0000 7f85c3234640 1 -- 172.21.15.106:0/1240999043 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc1a4a60 con 0x7f85bc100500 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.977+0000 7f85b9ffb640 1 -- 172.21.15.106:0/1240999043 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 23) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f85a80046c0 con 0x7f85bc100500 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.977+0000 7f85c5e19740 1 Processor -- start 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.977+0000 7f85c5e19740 1 -- start start 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.977+0000 7f85b9ffb640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f85ac0a2ab0 0x7f85ac0a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.977+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f72daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.977+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7368c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 0x55e71f7391f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85c5e19740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71f691b10 con 0x55e71f736e00 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85c5e19740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71f63a900 con 0x55e71f72e030 2024-03-28T02:11:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85c5e19740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71f68dbb0 con 0x55e71f72d710 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85b9ffb640 1 -- 172.21.15.106:0/1240999043 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f85a80cf770 con 0x7f85bc100500 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85c17d1640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f85ac0a2ab0 0x7f85ac0a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85c0fd0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f72daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7368c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:38004/0 (socket says 172.21.15.106:38004) 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85c0fd0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f72daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:57440/0 (socket says 172.21.15.106:57440) 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85bbfff640 1 -- 172.21.15.106:0/2241743584 learned_addr learned my addr 172.21.15.106:0/2241743584 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85c17d1640 1 --2- 172.21.15.106:0/2241743584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 0x55e71f7391f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85bbfff640 1 -- 172.21.15.106:0/2241743584 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 msgr2=0x55e71f72daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85bbfff640 1 --2- 172.21.15.106:0/2241743584 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f72daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85bbfff640 1 -- 172.21.15.106:0/2241743584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 msgr2=0x55e71f7391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85bbfff640 1 --2- 172.21.15.106:0/2241743584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 0x55e71f7391f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85bbfff640 1 -- 172.21.15.106:0/2241743584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e71f3a6210 con 0x55e71f72e030 2024-03-28T02:11:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85c17d1640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f85ac0a2ab0 0x7f85ac0a4f70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85b402c630 tx=0x7f85b4009950 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85bbfff640 1 --2- 172.21.15.106:0/2241743584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7368c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f85a800e9c0 tx=0x7f85a809b4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10120c48e1813a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85a6ffd640 1 -- 172.21.15.106:0/2241743584 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a80b4070 con 0x55e71f72e030 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85a6ffd640 1 -- 172.21.15.106:0/2241743584 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85a809fa30 con 0x55e71f72e030 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.978+0000 7f85a6ffd640 1 -- 172.21.15.106:0/2241743584 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a80ccd30 con 0x55e71f72e030 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 -- 172.21.15.106:0/2241743584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 msgr2=0x55e71f7368c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 --2- 172.21.15.106:0/2241743584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7368c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f85a800e9c0 tx=0x7f85a809b4e0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 -- 172.21.15.106:0/2241743584 shutdown_connections 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 --2- 172.21.15.106:0/2241743584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 0x55e71f7391f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 --2- 172.21.15.106:0/2241743584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7368c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 --2- 172.21.15.106:0/2241743584 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f72daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 -- 172.21.15.106:0/2241743584 >> 172.21.15.106:0/2241743584 conn(0x55e71f728030 msgr2=0x55e71f728410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 -- 172.21.15.106:0/2241743584 shutdown_connections 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 -- 172.21.15.106:0/2241743584 wait complete. 2024-03-28T02:11:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 -- start start 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.979+0000 7f85c5e19740 1 Processor -- start 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f733f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f734460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 0x55e71f7349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c5e19740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71f739a30 con 0x55e71f736e00 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c5e19740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71f64dd50 con 0x55e71f72e030 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c5e19740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71f739bb0 con 0x55e71f72d710 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c5e19740 10 client.0 _ll_register_callbacks cb 0x55e71f64d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c0fd0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f733f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c17d1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 0x55e71f7349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c0fd0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f733f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:57452/0 (socket says 172.21.15.106:57452) 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f734460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c0fd0640 1 -- 172.21.15.106:0/1365441842 learned_addr learned my addr 172.21.15.106:0/1365441842 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c17d1640 1 -- 172.21.15.106:0/1365441842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 msgr2=0x55e71f733f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c17d1640 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f733f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c17d1640 1 -- 172.21.15.106:0/1365441842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 msgr2=0x55e71f734460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c17d1640 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f734460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c17d1640 1 -- 172.21.15.106:0/1365441842 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e71f472620 con 0x55e71f736e00 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f85c17d1640 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 0x55e71f7349a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f85b4037040 tx=0x7f85b4005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e3c3510acb3e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.980+0000 7f859bfff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f859bfff640 1 -- 172.21.15.106:0/1365441842 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4051060 con 0x55e71f736e00 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f859bfff640 1 -- 172.21.15.106:0/1365441842 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f85b4004040 con 0x55e71f736e00 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c5e19740 1 -- 172.21.15.106:0/1365441842 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f85a809fa30 con 0x55e71f736e00 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f859bfff640 1 -- 172.21.15.106:0/1365441842 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85b4058050 con 0x55e71f736e00 2024-03-28T02:11:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c5e19740 10 client.15654 fetch_fsmap learned FSMap version 29 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c5e19740 1 -- 172.21.15.106:0/1365441842 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e71f745d80 con 0x55e71f736e00 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f859bfff640 1 -- 172.21.15.106:0/1365441842 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85a809fa30 con 0x55e71f736e00 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f859bfff640 1 -- 172.21.15.106:0/1365441842 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f85b404b070 con 0x55e71f736e00 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c5e19740 10 client.15654 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c5e19740 10 client.15654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c5e19740 20 client.15654 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c5e19740 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e71f747240 0x55e71f749630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c5e19740 4 client.15654 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c5e19740 1 -- 172.21.15.106:0/1365441842 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e71f69f050 con 0x55e71f747240 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.981+0000 7f85c0fd0640 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e71f747240 0x55e71f749630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.982+0000 7f859bfff640 10 client.15654 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.982+0000 7f85c0fd0640 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e71f747240 0x55e71f749630 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfce353e76dd3180 server_cookie=e1c1a7bddad42efb in_seq=0 out_seq=0 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.982+0000 7f859bfff640 1 -- 172.21.15.106:0/1365441842 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e71f69f050 con 0x55e71f747240 2024-03-28T02:11:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.982+0000 7f859bfff640 10 client.15654 handle_command_reply: tid=0 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.982+0000 7f85c5e19740 1 client.15654 shutdown 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 -- 172.21.15.106:0/1365441842 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 msgr2=0x55e71f7349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 0x55e71f7349a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f85b4037040 tx=0x7f85b4005d80 comp rx=0 tx=0).stop 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 -- 172.21.15.106:0/1365441842 shutdown_connections 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e71f747240 0x55e71f749630 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f736e00 0x55e71f7349a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f859bfff640 0 client.15654 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f734460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 --2- 172.21.15.106:0/1365441842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72d710 0x55e71f733f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 -- 172.21.15.106:0/1365441842 >> 172.21.15.106:0/1365441842 conn(0x55e71f728030 msgr2=0x55e71f729d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 -- 172.21.15.106:0/1365441842 shutdown_connections 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 1 -- 172.21.15.106:0/1365441842 wait complete. 2024-03-28T02:11:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.983+0000 7f85c5e19740 20 client.15654 trim_cache size 0 max 16384 2024-03-28T02:11:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c5e19740 1 Processor -- start 2024-03-28T02:11:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c5e19740 1 -- start start 2024-03-28T02:11:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f736650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f736e00 0x55e71f742800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c5e19740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1ae900 con 0x55e71f72d710 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c5e19740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1b0280 con 0x55e71f72e030 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c5e19740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1aef60 con 0x55e71f736e00 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c0fd0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f736650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c0fd0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f736650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50072/0 (socket says 172.21.15.106:50072) 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c0fd0640 1 -- 172.21.15.106:0/3021372993 learned_addr learned my addr 172.21.15.106:0/3021372993 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85bbfff640 1 --2- 172.21.15.106:0/3021372993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c17d1640 1 --2- 172.21.15.106:0/3021372993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f736e00 0x55e71f742800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c0fd0640 1 -- 172.21.15.106:0/3021372993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f736e00 msgr2=0x55e71f742800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c0fd0640 1 --2- 172.21.15.106:0/3021372993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f736e00 0x55e71f742800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c0fd0640 1 -- 172.21.15.106:0/3021372993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 msgr2=0x55e71f7422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c0fd0640 1 --2- 172.21.15.106:0/3021372993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.997+0000 7f85c0fd0640 1 -- 172.21.15.106:0/3021372993 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e71f749bf0 con 0x55e71f72d710 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c0fd0640 1 --2- 172.21.15.106:0/3021372993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f736650 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x55e71f749e20 tx=0x7f85b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62029e192baa2b01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85a67fc640 1 -- 172.21.15.106:0/3021372993 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0010630 con 0x55e71f72d710 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85a67fc640 1 -- 172.21.15.106:0/3021372993 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85b0026070 con 0x55e71f72d710 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85a67fc640 1 -- 172.21.15.106:0/3021372993 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b001ebc0 con 0x55e71f72d710 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c5e19740 1 -- 172.21.15.106:0/3021372993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 msgr2=0x55e71f736650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3021372993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f736650 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x55e71f749e20 tx=0x7f85b00046c0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c5e19740 1 -- 172.21.15.106:0/3021372993 shutdown_connections 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3021372993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f736e00 0x55e71f742800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3021372993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f72e030 0x55e71f7422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3021372993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f736650 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c5e19740 1 -- 172.21.15.106:0/3021372993 >> 172.21.15.106:0/3021372993 conn(0x55e71f728030 msgr2=0x55e71f727020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c5e19740 1 -- 172.21.15.106:0/3021372993 shutdown_connections 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.998+0000 7f85c5e19740 1 -- 172.21.15.106:0/3021372993 wait complete. 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c5e19740 1 -- start start 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c5e19740 1 Processor -- start 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f73e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72e030 0x55e71f731150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c5e19740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f736e00 0x55e71f731690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c5e19740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1afc20 con 0x55e71f72d710 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c5e19740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1af5c0 con 0x55e71f736e00 2024-03-28T02:11:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c5e19740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1acac0 con 0x55e71f72e030 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c5e19740 10 client.0 _ll_register_callbacks cb 0x7f85b4005f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c0fd0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f73e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c0fd0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f73e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:50086/0 (socket says 172.21.15.106:50086) 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c0fd0640 1 -- 172.21.15.106:0/3249265261 learned_addr learned my addr 172.21.15.106:0/3249265261 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c17d1640 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f736e00 0x55e71f731690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85bbfff640 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72e030 0x55e71f731150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c0fd0640 1 -- 172.21.15.106:0/3249265261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72e030 msgr2=0x55e71f731150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c0fd0640 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72e030 0x55e71f731150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c0fd0640 1 -- 172.21.15.106:0/3249265261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f736e00 msgr2=0x55e71f731690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c0fd0640 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f736e00 0x55e71f731690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:44.999+0000 7f85c0fd0640 1 -- 172.21.15.106:0/3249265261 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e71f731cd0 con 0x55e71f72d710 2024-03-28T02:11:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c0fd0640 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f73e9d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f85b0023040 tx=0x7f85b001e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c9c2b068f60761 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f859bfff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f859bfff640 1 -- 172.21.15.106:0/3249265261 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b002e020 con 0x55e71f72d710 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f859bfff640 1 -- 172.21.15.106:0/3249265261 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f85b001eb00 con 0x55e71f72d710 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c5e19740 1 -- 172.21.15.106:0/3249265261 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f85b0026070 con 0x55e71f72d710 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f859bfff640 1 -- 172.21.15.106:0/3249265261 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85b0039050 con 0x55e71f72d710 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c5e19740 10 client.15660 fetch_fsmap learned FSMap version 29 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c5e19740 1 -- 172.21.15.106:0/3249265261 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e71f7a3890 con 0x55e71f72d710 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f859bfff640 1 -- 172.21.15.106:0/3249265261 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85b0026070 con 0x55e71f72d710 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f859bfff640 1 -- 172.21.15.106:0/3249265261 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f85b002a430 con 0x55e71f72d710 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c5e19740 10 client.15660 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c5e19740 10 client.15660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c5e19740 20 client.15660 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e71f7a3ed0 0x55e71f7a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c5e19740 4 client.15660 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:11:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.000+0000 7f85c5e19740 1 -- 172.21.15.106:0/3249265261 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e71f6993a0 con 0x55e71f7a3ed0 2024-03-28T02:11:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.001+0000 7f85bbfff640 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e71f7a3ed0 0x55e71f7a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.001+0000 7f859bfff640 10 client.15660 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.001+0000 7f85bbfff640 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e71f7a3ed0 0x55e71f7a62f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=944fb9c7ab21be6 server_cookie=f0d869cfdcb7cf4f in_seq=0 out_seq=0 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f859bfff640 1 -- 172.21.15.106:0/3249265261 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e71f6993a0 con 0x55e71f7a3ed0 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f859bfff640 10 client.15660 handle_command_reply: tid=0 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 client.15660 shutdown 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 -- 172.21.15.106:0/3249265261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 msgr2=0x55e71f73e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f73e9d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f85b0023040 tx=0x7f85b001e760 comp rx=0 tx=0).stop 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 -- 172.21.15.106:0/3249265261 shutdown_connections 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e71f7a3ed0 0x55e71f7a62f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e71f736e00 0x55e71f731690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f859bfff640 0 client.15660 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e71f72e030 0x55e71f731150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 --2- 172.21.15.106:0/3249265261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e71f72d710 0x55e71f73e9d0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 -- 172.21.15.106:0/3249265261 >> 172.21.15.106:0/3249265261 conn(0x55e71f728030 msgr2=0x55e71f733ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 -- 172.21.15.106:0/3249265261 shutdown_connections 2024-03-28T02:11:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 1 -- 172.21.15.106:0/3249265261 wait complete. 2024-03-28T02:11:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c5e19740 20 client.15660 trim_cache size 0 max 16384 2024-03-28T02:11:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:11:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:11:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:11:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:11:45.003 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c3234640 1 -- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f85ac0a2ab0 msgr2=0x7f85ac0a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:45.003 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c3234640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f85ac0a2ab0 0x7f85ac0a4f70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85b402c630 tx=0x7f85b4009950 comp rx=0 tx=0).stop 2024-03-28T02:11:45.003 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c3234640 1 -- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 msgr2=0x7f85bc198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:45.003 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c3234640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 0x7f85bc198430 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f85a800e9f0 tx=0x7f85a800edf0 comp rx=0 tx=0).stop 2024-03-28T02:11:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c3234640 1 -- 172.21.15.106:0/1240999043 shutdown_connections 2024-03-28T02:11:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c3234640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f85ac0a2ab0 0x7f85ac0a4f70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c3234640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85bc107660 0x7f85bc19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c3234640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc100500 0x7f85bc198430 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.004+0000 7f85c3234640 1 --2- 172.21.15.106:0/1240999043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc0ffb40 0x7f85bc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.005+0000 7f85c3234640 1 -- 172.21.15.106:0/1240999043 >> 172.21.15.106:0/1240999043 conn(0x7f85bc076ac0 msgr2=0x7f85bc104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.005+0000 7f85c3234640 1 -- 172.21.15.106:0/1240999043 shutdown_connections 2024-03-28T02:11:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:45.005+0000 7f85c3234640 1 -- 172.21.15.106:0/1240999043 wait complete. 2024-03-28T02:11:45.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:45 smithi153 ceph-mon[36228]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T02:11:45.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:45 smithi106 ceph-mon[31965]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T02:11:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:45 smithi178 ceph-mon[36521]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T02:11:47.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:47 smithi106 ceph-mon[31965]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:11:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:47 smithi178 ceph-mon[36521]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:11:47.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:47 smithi153 ceph-mon[36228]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:11:49.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:49 smithi106 ceph-mon[31965]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:11:49.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:49 smithi178 ceph-mon[36521]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:11:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:49 smithi153 ceph-mon[36228]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:11:51.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:51 smithi106 ceph-mon[31965]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:11:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:51 smithi178 ceph-mon[36521]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:11:51.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:51 smithi153 ceph-mon[36228]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:11:53.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:53 smithi153 ceph-mon[36228]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-28T02:11:53.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:53 smithi106 ceph-mon[31965]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-28T02:11:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:53 smithi178 ceph-mon[36521]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-28T02:11:55.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:55 smithi106 ceph-mon[31965]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-28T02:11:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:55 smithi178 ceph-mon[36521]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-28T02:11:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:55 smithi153 ceph-mon[36228]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-28T02:11:57.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:57 smithi106 ceph-mon[31965]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-28T02:11:57.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:57 smithi178 ceph-mon[36521]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-28T02:11:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:57 smithi153 ceph-mon[36228]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-28T02:11:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.099+0000 7fe8caff3640 1 -- 172.21.15.106:0/4053025960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4100570 msgr2=0x7fe8c41009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.099+0000 7fe8caff3640 1 --2- 172.21.15.106:0/4053025960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4100570 0x7fe8c41009f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009900 tx=0x7fe8b402bc30 comp rx=0 tx=0).stop 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.099+0000 7fe8caff3640 1 -- 172.21.15.106:0/4053025960 shutdown_connections 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.099+0000 7fe8caff3640 1 --2- 172.21.15.106:0/4053025960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8c4100f30 0x7fe8c410d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.099+0000 7fe8caff3640 1 --2- 172.21.15.106:0/4053025960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4100570 0x7fe8c41009f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.099+0000 7fe8caff3640 1 --2- 172.21.15.106:0/4053025960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8c4106700 0x7fe8c4106ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.099+0000 7fe8caff3640 1 -- 172.21.15.106:0/4053025960 >> 172.21.15.106:0/4053025960 conn(0x7fe8c40776d0 msgr2=0x7fe8c4077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.099+0000 7fe8caff3640 1 -- 172.21.15.106:0/4053025960 shutdown_connections 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.099+0000 7fe8caff3640 1 -- 172.21.15.106:0/4053025960 wait complete. 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8caff3640 1 Processor -- start 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8caff3640 1 -- start start 2024-03-28T02:11:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8caff3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8c4100570 0x7fe8c4197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8caff3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8c4100f30 0x7fe8c4198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8caff3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4106700 0x7fe8c419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8caff3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c4111a90 con 0x7fe8c4106700 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8caff3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c4111910 con 0x7fe8c4100f30 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8caff3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c4111c10 con 0x7fe8c4100570 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8c9590640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4106700 0x7fe8c419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8c9590640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4106700 0x7fe8c419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:59074/0 (socket says 172.21.15.106:59074) 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8c9590640 1 -- 172.21.15.106:0/603332743 learned_addr learned my addr 172.21.15.106:0/603332743 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8c8d8f640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8c4100570 0x7fe8c4197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.100+0000 7fe8c3fff640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8c4100f30 0x7fe8c4198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8c9590640 1 -- 172.21.15.106:0/603332743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8c4100570 msgr2=0x7fe8c4197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8c9590640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8c4100570 0x7fe8c4197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8c9590640 1 -- 172.21.15.106:0/603332743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8c4100f30 msgr2=0x7fe8c4198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8c9590640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8c4100f30 0x7fe8c4198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8c9590640 1 -- 172.21.15.106:0/603332743 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c419cde0 con 0x7fe8c4106700 2024-03-28T02:11:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8c9590640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4106700 0x7fe8c419c6b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00e970 tx=0x7fe8ac00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8c1ffb640 1 -- 172.21.15.106:0/603332743 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac00ccc0 con 0x7fe8c4106700 2024-03-28T02:11:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8caff3640 1 -- 172.21.15.106:0/603332743 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c419d0d0 con 0x7fe8c4106700 2024-03-28T02:11:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8c1ffb640 1 -- 172.21.15.106:0/603332743 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac00ce60 con 0x7fe8c4106700 2024-03-28T02:11:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8c1ffb640 1 -- 172.21.15.106:0/603332743 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac010450 con 0x7fe8c4106700 2024-03-28T02:11:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.101+0000 7fe8caff3640 1 -- 172.21.15.106:0/603332743 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c41a4960 con 0x7fe8c4106700 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.103+0000 7fe8c1ffb640 1 -- 172.21.15.106:0/603332743 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 23) v1 ==== 137467+0+0 (secure 0 0 0) 0x7fe8ac005020 con 0x7fe8c4106700 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8c1ffb640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8b00a2ab0 0x7fe8b00a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8cdbdc740 1 Processor -- start 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8cdbdc740 1 -- start start 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8c8d8f640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8b00a2ab0 0x7fe8b00a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8c1ffb640 1 -- 172.21.15.106:0/603332743 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fe8ac0cf780 con 0x7fe8c4106700 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e80af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 0x558589e898c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 0x558589e8c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x558589de4b10 con 0x558589e80710 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x558589d8d900 con 0x558589e81030 2024-03-28T02:11:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558589de0bb0 con 0x558589e89e00 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8c8d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e80af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8c8d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e80af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:59090/0 (socket says 172.21.15.106:59090) 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8c8d8f640 1 -- 172.21.15.106:0/3977629916 learned_addr learned my addr 172.21.15.106:0/3977629916 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8c9590640 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 0x558589e8c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.104+0000 7fe8c3fff640 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 0x558589e898c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8c9590640 1 -- 172.21.15.106:0/3977629916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 msgr2=0x558589e898c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8c9590640 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 0x558589e898c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8c9590640 1 -- 172.21.15.106:0/3977629916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 msgr2=0x558589e80af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8c9590640 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e80af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8c8d8f640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8b00a2ab0 0x7fe8b00a4f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc0138b0 tx=0x7fe8bc013cb0 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8c9590640 1 -- 172.21.15.106:0/3977629916 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558589af9210 con 0x558589e89e00 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8c3fff640 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 0x558589e898c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8c9590640 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 0x558589e8c1f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0cca90 tx=0x7fe8ac09b560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0a3171c03fbcb10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8baffd640 1 -- 172.21.15.106:0/3977629916 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac09fa40 con 0x558589e89e00 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8baffd640 1 -- 172.21.15.106:0/3977629916 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac0a4090 con 0x558589e89e00 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.105+0000 7fe8baffd640 1 -- 172.21.15.106:0/3977629916 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0860e0 con 0x558589e89e00 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/3977629916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 msgr2=0x558589e8c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 0x558589e8c1f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0cca90 tx=0x7fe8ac09b560 comp rx=0 tx=0).stop 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/3977629916 shutdown_connections 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 0x558589e8c1f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 0x558589e898c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/3977629916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e80af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/3977629916 >> 172.21.15.106:0/3977629916 conn(0x558589e7b030 msgr2=0x558589e7b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/3977629916 shutdown_connections 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/3977629916 wait complete. 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 -- start start 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.106+0000 7fe8cdbdc740 1 Processor -- start 2024-03-28T02:11:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e86e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 0x558589e87370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e89e00 0x558589e878b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x558589e8c9d0 con 0x558589e80710 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x558589da0d50 con 0x558589e89e00 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558589e8cb50 con 0x558589e81030 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8cdbdc740 10 client.0 _ll_register_callbacks cb 0x558589da0720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c8d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e86e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c8d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e86e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:59092/0 (socket says 172.21.15.106:59092) 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c8d8f640 1 -- 172.21.15.106:0/2067051148 learned_addr learned my addr 172.21.15.106:0/2067051148 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c3fff640 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 0x558589e87370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c9590640 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e89e00 0x558589e878b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c8d8f640 1 -- 172.21.15.106:0/2067051148 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 msgr2=0x558589e87370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c8d8f640 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 0x558589e87370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c8d8f640 1 -- 172.21.15.106:0/2067051148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e89e00 msgr2=0x558589e878b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c8d8f640 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e89e00 0x558589e878b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c8d8f640 1 -- 172.21.15.106:0/2067051148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558589bc5620 con 0x558589e80710 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8c8d8f640 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e86e30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc003c00 tx=0x7fe8bc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be412a5bd78622f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8a3fff640 1 -- 172.21.15.106:0/2067051148 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc012270 con 0x558589e80710 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8a3fff640 1 -- 172.21.15.106:0/2067051148 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fe8bc010430 con 0x558589e80710 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.107+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2067051148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe8ac0a4090 con 0x558589e80710 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8a3fff640 1 -- 172.21.15.106:0/2067051148 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8bc031050 con 0x558589e80710 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8cdbdc740 10 client.15672 fetch_fsmap learned FSMap version 29 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2067051148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x558589e98c30 con 0x558589e80710 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8a3fff640 1 -- 172.21.15.106:0/2067051148 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac0a4090 con 0x558589e80710 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8a3fff640 1 -- 172.21.15.106:0/2067051148 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fe8bc01f070 con 0x558589e80710 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8cdbdc740 10 client.15672 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8cdbdc740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:11:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8cdbdc740 20 client.15672 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x558589e9a0f0 0x558589e9c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8cdbdc740 4 client.15672 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2067051148 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558589df2050 con 0x558589e9a0f0 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.108+0000 7fe8c3fff640 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x558589e9a0f0 0x558589e9c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8a3fff640 10 client.15672 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8c3fff640 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x558589e9a0f0 0x558589e9c4e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=464d14fdc407187c server_cookie=39bb614692f1f065 in_seq=0 out_seq=0 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8a3fff640 1 -- 172.21.15.106:0/2067051148 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558589df2050 con 0x558589e9a0f0 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8a3fff640 10 client.15672 handle_command_reply: tid=0 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 client.15672 shutdown 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2067051148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 msgr2=0x558589e86e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e86e30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc003c00 tx=0x7fe8bc004ab0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2067051148 shutdown_connections 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x558589e9a0f0 0x558589e9c4e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e89e00 0x558589e878b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 0x558589e87370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8a3fff640 0 client.15672 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2067051148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e86e30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2067051148 >> 172.21.15.106:0/2067051148 conn(0x558589e7b030 msgr2=0x558589e7cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2067051148 shutdown_connections 2024-03-28T02:11:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.109+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2067051148 wait complete. 2024-03-28T02:11:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.110+0000 7fe8cdbdc740 20 client.15672 trim_cache size 0 max 16384 2024-03-28T02:11:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8cdbdc740 1 Processor -- start 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8cdbdc740 1 -- start start 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e80710 0x558589e89560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 0x558589e79ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e89e00 0x558589e95260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c41ae610 con 0x558589e89e00 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c41aff90 con 0x558589e80710 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c41aec70 con 0x558589e81030 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8c9590640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e89e00 0x558589e95260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8c9590640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e89e00 0x558589e95260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:59104/0 (socket says 172.21.15.106:59104) 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8c3fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 0x558589e79ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8c9590640 1 -- 172.21.15.106:0/2982674534 learned_addr learned my addr 172.21.15.106:0/2982674534 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.123+0000 7fe8c8d8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e80710 0x558589e89560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8c9590640 1 -- 172.21.15.106:0/2982674534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 msgr2=0x558589e79ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8c9590640 1 --2- 172.21.15.106:0/2982674534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 0x558589e79ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8c9590640 1 -- 172.21.15.106:0/2982674534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e80710 msgr2=0x558589e89560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8c9590640 1 --2- 172.21.15.106:0/2982674534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e80710 0x558589e89560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8c9590640 1 -- 172.21.15.106:0/2982674534 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558589e80e60 con 0x558589e89e00 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8c9590640 1 --2- 172.21.15.106:0/2982674534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e89e00 0x558589e95260 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0cd150 tx=0x7fe8ac004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45317645cfd30e29 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8ba7fc640 1 -- 172.21.15.106:0/2982674534 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0b8070 con 0x558589e89e00 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8ba7fc640 1 -- 172.21.15.106:0/2982674534 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac0a4090 con 0x558589e89e00 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8ba7fc640 1 -- 172.21.15.106:0/2982674534 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0ccd40 con 0x558589e89e00 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2982674534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e89e00 msgr2=0x558589e95260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.124+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2982674534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e89e00 0x558589e95260 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0cd150 tx=0x7fe8ac004a50 comp rx=0 tx=0).stop 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2982674534 shutdown_connections 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2982674534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e89e00 0x558589e95260 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2982674534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e81030 0x558589e79ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/2982674534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e80710 0x558589e89560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2982674534 >> 172.21.15.106:0/2982674534 conn(0x558589e7b030 msgr2=0x558589e79790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2982674534 shutdown_connections 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/2982674534 wait complete. 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 -- start start 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 Processor -- start 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e91890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 0x558589e8dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 0x558589e8e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c41af930 con 0x558589e80710 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c41af2d0 con 0x558589e81030 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c41ac7d0 con 0x558589e89e00 2024-03-28T02:11:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8cdbdc740 10 client.0 _ll_register_callbacks cb 0x7fe8bc00fee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8c8d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e91890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8c8d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e91890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:59110/0 (socket says 172.21.15.106:59110) 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8c8d8f640 1 -- 172.21.15.106:0/4148746765 learned_addr learned my addr 172.21.15.106:0/4148746765 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.125+0000 7fe8c9590640 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 0x558589e8e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8c3fff640 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 0x558589e8dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8c8d8f640 1 -- 172.21.15.106:0/4148746765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 msgr2=0x558589e8e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8c8d8f640 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 0x558589e8e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8c8d8f640 1 -- 172.21.15.106:0/4148746765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 msgr2=0x558589e8dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8c8d8f640 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 0x558589e8dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8c8d8f640 1 -- 172.21.15.106:0/4148746765 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558589e8e690 con 0x558589e80710 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8c8d8f640 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e91890 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc00f790 tx=0x7fe8bc020800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3963a7da95dd5c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:11:58.123 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8a3fff640 1 -- 172.21.15.106:0/4148746765 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc009200 con 0x558589e80710 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8a3fff640 1 -- 172.21.15.106:0/4148746765 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fe8bc004b20 con 0x558589e80710 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/4148746765 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe8ac0a4090 con 0x558589e80710 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8a3fff640 1 -- 172.21.15.106:0/4148746765 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8bc042050 con 0x558589e80710 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8cdbdc740 10 client.15678 fetch_fsmap learned FSMap version 29 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/4148746765 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x558589efaa70 con 0x558589e80710 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8a3fff640 1 -- 172.21.15.106:0/4148746765 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac0a4090 con 0x558589e80710 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.126+0000 7fe8a3fff640 1 -- 172.21.15.106:0/4148746765 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fe8bc034070 con 0x558589e80710 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.127+0000 7fe8cdbdc740 10 client.15678 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.127+0000 7fe8cdbdc740 10 client.15678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.127+0000 7fe8cdbdc740 20 client.15678 populate_metadata read hostname 'smithi106' 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.127+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x558589efb1c0 0x558589efd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.127+0000 7fe8cdbdc740 4 client.15678 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.127+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/4148746765 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558589dec3a0 con 0x558589efb1c0 2024-03-28T02:11:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.127+0000 7fe8c3fff640 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x558589efb1c0 0x558589efd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:11:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.127+0000 7fe8c3fff640 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x558589efb1c0 0x558589efd5e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43dde64caaff03d3 server_cookie=9aed606740f53ee1 in_seq=0 out_seq=0 2024-03-28T02:11:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.127+0000 7fe8a3fff640 10 client.15678 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8a3fff640 1 -- 172.21.15.106:0/4148746765 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558589dec3a0 con 0x558589efb1c0 2024-03-28T02:11:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8a3fff640 10 client.15678 handle_command_reply: tid=0 2024-03-28T02:11:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 client.15678 shutdown 2024-03-28T02:11:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/4148746765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 msgr2=0x558589e91890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e91890 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc00f790 tx=0x7fe8bc020800 comp rx=0 tx=0).stop 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/4148746765 shutdown_connections 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x558589efb1c0 0x558589efd5e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558589e89e00 0x558589e8e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8a3fff640 0 client.15678 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558589e81030 0x558589e8dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 --2- 172.21.15.106:0/4148746765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558589e80710 0x558589e91890 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/4148746765 >> 172.21.15.106:0/4148746765 conn(0x558589e7b030 msgr2=0x558589e87680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/4148746765 shutdown_connections 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.133+0000 7fe8cdbdc740 1 -- 172.21.15.106:0/4148746765 wait complete. 2024-03-28T02:11:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8cdbdc740 20 client.15678 trim_cache size 0 max 16384 2024-03-28T02:11:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:11:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:11:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:11:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 -- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8b00a2ab0 msgr2=0x7fe8b00a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8b00a2ab0 0x7fe8b00a4f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc0138b0 tx=0x7fe8bc013cb0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 -- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4106700 msgr2=0x7fe8c419c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4106700 0x7fe8c419c6b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00e970 tx=0x7fe8ac00ed70 comp rx=0 tx=0).stop 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 -- 172.21.15.106:0/603332743 shutdown_connections 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8b00a2ab0 0x7fe8b00a4f70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8c4106700 0x7fe8c419c6b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8c4100f30 0x7fe8c4198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 --2- 172.21.15.106:0/603332743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8c4100570 0x7fe8c4197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:11:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 -- 172.21.15.106:0/603332743 >> 172.21.15.106:0/603332743 conn(0x7fe8c40776d0 msgr2=0x7fe8c40feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:11:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 -- 172.21.15.106:0/603332743 shutdown_connections 2024-03-28T02:11:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:11:58.134+0000 7fe8caff3640 1 -- 172.21.15.106:0/603332743 wait complete. 2024-03-28T02:11:59.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:11:59 smithi106 ceph-mon[31965]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-28T02:11:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:11:59 smithi178 ceph-mon[36521]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-28T02:11:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:11:59 smithi153 ceph-mon[36228]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-28T02:12:01.332 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:12:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:12:00.893+0000 7f1122061700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:12:01.333 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:12:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:12:00.893+0000 7f1122061700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:12:01.333 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:12:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:12:00.893+0000 7f1122061700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:12:01.333 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:12:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:12:00.893+0000 7f1122061700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:12:01.333 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:12:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:12:00.893+0000 7f1122061700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T02:12:01.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:01 smithi106 ceph-mon[31965]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T02:12:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:01 smithi178 ceph-mon[36521]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T02:12:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:01 smithi153 ceph-mon[36228]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T02:12:03.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:03 smithi106 ceph-mon[31965]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-28T02:12:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:03 smithi178 ceph-mon[36521]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-28T02:12:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:03 smithi153 ceph-mon[36228]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-28T02:12:04.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:04 smithi106 ceph-mon[31965]: mgrmap e24: x(active, since 47s), standbys: z, y 2024-03-28T02:12:04.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:04 smithi178 ceph-mon[36521]: mgrmap e24: x(active, since 47s), standbys: z, y 2024-03-28T02:12:04.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:04 smithi153 ceph-mon[36228]: mgrmap e24: x(active, since 47s), standbys: z, y 2024-03-28T02:12:05.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:05 smithi106 ceph-mon[31965]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T02:12:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:05 smithi178 ceph-mon[36521]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T02:12:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:05 smithi153 ceph-mon[36228]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T02:12:07.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:07 smithi106 ceph-mon[31965]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-28T02:12:07.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:07 smithi178 ceph-mon[36521]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-28T02:12:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:07 smithi153 ceph-mon[36228]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-28T02:12:09.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:09 smithi106 ceph-mon[31965]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T02:12:09.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:09 smithi178 ceph-mon[36521]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T02:12:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:09 smithi153 ceph-mon[36228]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T02:12:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 -- 172.21.15.106:0/16007626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8301065d0 msgr2=0x7fe8301069b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 --2- 172.21.15.106:0/16007626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8301065d0 0x7fe8301069b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe820009930 tx=0x7fe82002bc30 comp rx=0 tx=0).stop 2024-03-28T02:12:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 -- 172.21.15.106:0/16007626 shutdown_connections 2024-03-28T02:12:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 --2- 172.21.15.106:0/16007626 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe830101670 0x7fe83010d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 --2- 172.21.15.106:0/16007626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe830100cb0 0x7fe830101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 --2- 172.21.15.106:0/16007626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8301065d0 0x7fe8301069b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 -- 172.21.15.106:0/16007626 >> 172.21.15.106:0/16007626 conn(0x7fe830076b30 msgr2=0x7fe8300fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 -- 172.21.15.106:0/16007626 shutdown_connections 2024-03-28T02:12:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 -- 172.21.15.106:0/16007626 wait complete. 2024-03-28T02:12:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.228+0000 7fe8364f8640 1 Processor -- start 2024-03-28T02:12:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe8364f8640 1 -- start start 2024-03-28T02:12:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe8364f8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe830100cb0 0x7fe830198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe8364f8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe830101670 0x7fe830199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe8364f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8301065d0 0x7fe8301a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe8364f8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301108f0 con 0x7fe830100cb0 2024-03-28T02:12:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe8364f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830110770 con 0x7fe8301065d0 2024-03-28T02:12:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe8364f8640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830110a70 con 0x7fe830101670 2024-03-28T02:12:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe830100cb0 0x7fe830198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe830100cb0 0x7fe830198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37598/0 (socket says 172.21.15.106:37598) 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe82ffff640 1 -- 172.21.15.106:0/1911468643 learned_addr learned my addr 172.21.15.106:0/1911468643 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe834a95640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8301065d0 0x7fe8301a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe830101670 0x7fe830199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe82ffff640 1 -- 172.21.15.106:0/1911468643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe830101670 msgr2=0x7fe830199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe82ffff640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe830101670 0x7fe830199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe82ffff640 1 -- 172.21.15.106:0/1911468643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8301065d0 msgr2=0x7fe8301a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.229+0000 7fe82ffff640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8301065d0 0x7fe8301a3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.230+0000 7fe82ffff640 1 -- 172.21.15.106:0/1911468643 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8301a4210 con 0x7fe830100cb0 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.230+0000 7fe834a95640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8301065d0 0x7fe8301a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.230+0000 7fe82ffff640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe830100cb0 0x7fe830198c50 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe820009230 tx=0x7fe82002cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.230+0000 7fe82d7fa640 1 -- 172.21.15.106:0/1911468643 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820041070 con 0x7fe830100cb0 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.230+0000 7fe8364f8640 1 -- 172.21.15.106:0/1911468643 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8301a43e0 con 0x7fe830100cb0 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.230+0000 7fe82d7fa640 1 -- 172.21.15.106:0/1911468643 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe820004450 con 0x7fe830100cb0 2024-03-28T02:12:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.230+0000 7fe82d7fa640 1 -- 172.21.15.106:0/1911468643 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820003dc0 con 0x7fe830100cb0 2024-03-28T02:12:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.230+0000 7fe8364f8640 1 -- 172.21.15.106:0/1911468643 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8301a4850 con 0x7fe830100cb0 2024-03-28T02:12:11.227 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.233+0000 7fe82d7fa640 1 -- 172.21.15.106:0/1911468643 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 24) v1 ==== 137524+0+0 (secure 0 0 0) 0x7fe820044070 con 0x7fe830100cb0 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.233+0000 7fe8391c3740 1 Processor -- start 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.233+0000 7fe8391c3740 1 -- start start 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.233+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243067710 0x558243067af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.233+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243068030 0x5582430708c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.233+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x5582430731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.233+0000 7fe8391c3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x558242fcbb10 con 0x558243070e00 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.233+0000 7fe8391c3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x558242f74900 con 0x558243067710 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.233+0000 7fe8391c3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558242fc7bb0 con 0x558243068030 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe834a95640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243067710 0x558243067af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x5582430731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82d7fa640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8180a2b30 0x7fe8180a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe834a95640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243067710 0x558243067af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:54834/0 (socket says 172.21.15.106:54834) 2024-03-28T02:12:11.228 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe834a95640 1 -- 172.21.15.106:0/3699002207 learned_addr learned my addr 172.21.15.106:0/3699002207 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82ffff640 1 --2- 172.21.15.106:0/3699002207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243068030 0x5582430708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82f7fe640 1 -- 172.21.15.106:0/3699002207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243068030 msgr2=0x5582430708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/3699002207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243068030 0x5582430708c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82d7fa640 1 -- 172.21.15.106:0/1911468643 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fe8200f0e10 con 0x7fe830100cb0 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe834a95640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8180a2b30 0x7fe8180a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82f7fe640 1 -- 172.21.15.106:0/3699002207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243067710 msgr2=0x558243067af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/3699002207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243067710 0x558243067af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82f7fe640 1 -- 172.21.15.106:0/3699002207 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558242ce0210 con 0x558243070e00 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/3699002207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x5582430731f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe824009bf0 tx=0x7fe824011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efaee4ddc7b2dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe834a95640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8180a2b30 0x7fe8180a4ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe828005c40 tx=0x7fe828004ab0 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe8167fc640 1 -- 172.21.15.106:0/3699002207 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824004050 con 0x558243070e00 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.234+0000 7fe8167fc640 1 -- 172.21.15.106:0/3699002207 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe824005500 con 0x558243070e00 2024-03-28T02:12:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8167fc640 1 -- 172.21.15.106:0/3699002207 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824004e40 con 0x558243070e00 2024-03-28T02:12:11.233 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8391c3740 1 -- 172.21.15.106:0/3699002207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 msgr2=0x5582430731f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:11.233 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8391c3740 1 --2- 172.21.15.106:0/3699002207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x5582430731f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe824009bf0 tx=0x7fe824011810 comp rx=0 tx=0).stop 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8391c3740 1 -- 172.21.15.106:0/3699002207 shutdown_connections 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8391c3740 1 --2- 172.21.15.106:0/3699002207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x5582430731f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8391c3740 1 --2- 172.21.15.106:0/3699002207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243068030 0x5582430708c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8391c3740 1 --2- 172.21.15.106:0/3699002207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243067710 0x558243067af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8391c3740 1 -- 172.21.15.106:0/3699002207 >> 172.21.15.106:0/3699002207 conn(0x558243062030 msgr2=0x558243062410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8391c3740 1 -- 172.21.15.106:0/3699002207 shutdown_connections 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.235+0000 7fe8391c3740 1 -- 172.21.15.106:0/3699002207 wait complete. 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe8391c3740 1 -- start start 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe8391c3740 1 Processor -- start 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243067710 0x55824306de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 0x55824306e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243070e00 0x55824306e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe8391c3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x558243077e40 con 0x558243067710 2024-03-28T02:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe8391c3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x558242f87d50 con 0x558243068030 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe8391c3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558243077fc0 con 0x558243070e00 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe8391c3740 10 client.0 _ll_register_callbacks cb 0x558242f87720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243067710 0x55824306de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243067710 0x55824306de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37616/0 (socket says 172.21.15.106:37616) 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe82ffff640 1 -- 172.21.15.106:0/1375069808 learned_addr learned my addr 172.21.15.106:0/1375069808 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe834a95640 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243070e00 0x55824306e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.236+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 0x55824306e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe82ffff640 1 -- 172.21.15.106:0/1375069808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243070e00 msgr2=0x55824306e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe82ffff640 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243070e00 0x55824306e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe82ffff640 1 -- 172.21.15.106:0/1375069808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 msgr2=0x55824306e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe82ffff640 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 0x55824306e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe82ffff640 1 -- 172.21.15.106:0/1375069808 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558242dac620 con 0x558243067710 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe82ffff640 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243067710 0x55824306de60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe820006010 tx=0x7fe820046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c342e75fadd45c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe80f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:12:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe80f7fe640 1 -- 172.21.15.106:0/1375069808 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8200bd060 con 0x558243067710 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe80f7fe640 1 -- 172.21.15.106:0/1375069808 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fe8200b7bd0 con 0x558243067710 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe8391c3740 1 -- 172.21.15.106:0/1375069808 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe824005500 con 0x558243067710 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe80f7fe640 1 -- 172.21.15.106:0/1375069808 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8200c1100 con 0x558243067710 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe8391c3740 10 client.15696 fetch_fsmap learned FSMap version 29 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe8391c3740 1 -- 172.21.15.106:0/1375069808 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x558243084040 con 0x558243067710 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.237+0000 7fe80f7fe640 1 -- 172.21.15.106:0/1375069808 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe824005500 con 0x558243067710 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe80f7fe640 1 -- 172.21.15.106:0/1375069808 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fe8200ee340 con 0x558243067710 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe8391c3740 10 client.15696 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe8391c3740 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe8391c3740 20 client.15696 populate_metadata read hostname 'smithi106' 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5582430854f0 0x558243087920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe8391c3740 4 client.15696 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe8391c3740 1 -- 172.21.15.106:0/1375069808 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558242fd9050 con 0x5582430854f0 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5582430854f0 0x558243087920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.236 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe80f7fe640 10 client.15696 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.238+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5582430854f0 0x558243087920 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95989327516a24e9 server_cookie=5e64a450c930ea6 in_seq=0 out_seq=0 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe80f7fe640 1 -- 172.21.15.106:0/1375069808 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558242fd9050 con 0x5582430854f0 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe80f7fe640 10 client.15696 handle_command_reply: tid=0 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 client.15696 shutdown 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 -- 172.21.15.106:0/1375069808 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243067710 msgr2=0x55824306de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243067710 0x55824306de60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe820006010 tx=0x7fe820046000 comp rx=0 tx=0).stop 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 -- 172.21.15.106:0/1375069808 shutdown_connections 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5582430854f0 0x558243087920 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243070e00 0x55824306e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 0x55824306e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1375069808 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243067710 0x55824306de60 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 -- 172.21.15.106:0/1375069808 >> 172.21.15.106:0/1375069808 conn(0x558243062030 msgr2=0x558243063d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe80f7fe640 0 client.15696 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 -- 172.21.15.106:0/1375069808 shutdown_connections 2024-03-28T02:12:11.237 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 1 -- 172.21.15.106:0/1375069808 wait complete. 2024-03-28T02:12:11.238 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.239+0000 7fe8391c3740 20 client.15696 trim_cache size 0 max 16384 2024-03-28T02:12:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe8391c3740 1 Processor -- start 2024-03-28T02:12:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe8391c3740 1 -- start start 2024-03-28T02:12:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x558243070590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243068030 0x558243063b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243070e00 0x5582430806d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe8391c3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301ac4f0 con 0x558243068030 2024-03-28T02:12:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe8391c3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301ade70 con 0x558243070e00 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe8391c3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301acb50 con 0x558243067710 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe82f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243068030 0x558243063b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe82f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243068030 0x558243063b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:37622/0 (socket says 172.21.15.106:37622) 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe82f7fe640 1 -- 172.21.15.106:0/1580608031 learned_addr learned my addr 172.21.15.106:0/1580608031 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe82ffff640 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x558243070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.253+0000 7fe834a95640 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243070e00 0x5582430806d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe82f7fe640 1 -- 172.21.15.106:0/1580608031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 msgr2=0x558243070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x558243070590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe82f7fe640 1 -- 172.21.15.106:0/1580608031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243070e00 msgr2=0x5582430806d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243070e00 0x5582430806d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe82f7fe640 1 -- 172.21.15.106:0/1580608031 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558243067e60 con 0x558243068030 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe82ffff640 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x558243070590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243068030 0x558243063b90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe824009b40 tx=0x7fe824009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d448fa9b35596db1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:11.248 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe815ffb640 1 -- 172.21.15.106:0/1580608031 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82402e280 con 0x558243068030 2024-03-28T02:12:11.249 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe815ffb640 1 -- 172.21.15.106:0/1580608031 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe824004530 con 0x558243068030 2024-03-28T02:12:11.249 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.254+0000 7fe815ffb640 1 -- 172.21.15.106:0/1580608031 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824003cc0 con 0x558243068030 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 -- 172.21.15.106:0/1580608031 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243068030 msgr2=0x558243063b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243068030 0x558243063b90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe824009b40 tx=0x7fe824009680 comp rx=0 tx=0).stop 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 -- 172.21.15.106:0/1580608031 shutdown_connections 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243070e00 0x5582430806d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243068030 0x558243063b90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 --2- 172.21.15.106:0/1580608031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x558243070590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 -- 172.21.15.106:0/1580608031 >> 172.21.15.106:0/1580608031 conn(0x558243062030 msgr2=0x558243063480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 -- 172.21.15.106:0/1580608031 shutdown_connections 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 -- 172.21.15.106:0/1580608031 wait complete. 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.255+0000 7fe8391c3740 1 -- start start 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe8391c3740 1 Processor -- start 2024-03-28T02:12:11.252 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x55824306a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 0x55824306a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe8391c3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x55824306ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe8391c3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301ad810 con 0x558243070e00 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe8391c3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301ad1b0 con 0x558243068030 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe8391c3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301aa6b0 con 0x558243067710 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe8391c3740 10 client.0 _ll_register_callbacks cb 0x7fe8200095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x55824306a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe834a95640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x55824306ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x55824306a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:55870/0 (socket says 172.21.15.106:55870) 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe82ffff640 1 -- 172.21.15.106:0/512335521 learned_addr learned my addr 172.21.15.106:0/512335521 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 0x55824306a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe82ffff640 1 -- 172.21.15.106:0/512335521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 msgr2=0x55824306a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe82ffff640 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 0x55824306a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe82ffff640 1 -- 172.21.15.106:0/512335521 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 msgr2=0x55824306ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:11.253 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe82ffff640 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x55824306ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.256+0000 7fe82ffff640 1 -- 172.21.15.106:0/512335521 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55824306bea0 con 0x558243067710 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe834a95640 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x55824306ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe82ffff640 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x55824306a2d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe82002c820 tx=0x7fe8200bc500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45a61eada3b8f4b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe80f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.178:3300/0 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe80f7fe640 1 -- 172.21.15.106:0/512335521 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8200bcba0 con 0x558243067710 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe80f7fe640 1 -- 172.21.15.106:0/512335521 <== mon.1 v2:172.21.15.178:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fe8200b7bd0 con 0x558243067710 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe8391c3740 1 -- 172.21.15.106:0/512335521 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe824004530 con 0x558243067710 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe80f7fe640 1 -- 172.21.15.106:0/512335521 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8200ee0b0 con 0x558243067710 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe8391c3740 10 client.25477 fetch_fsmap learned FSMap version 29 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe8391c3740 1 -- 172.21.15.106:0/512335521 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55824306bea0 con 0x558243067710 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe80f7fe640 1 -- 172.21.15.106:0/512335521 <== mon.1 v2:172.21.15.178:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe824004530 con 0x558243067710 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe80f7fe640 1 -- 172.21.15.106:0/512335521 <== mon.1 v2:172.21.15.178:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fe8200dbab0 con 0x558243067710 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe8391c3740 10 client.25477 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:12:11.254 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.257+0000 7fe8391c3740 10 client.25477 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:12:11.255 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.258+0000 7fe8391c3740 20 client.25477 populate_metadata read hostname 'smithi106' 2024-03-28T02:12:11.255 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.258+0000 7fe8391c3740 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5582430e1160 0x5582430e3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:11.255 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.258+0000 7fe8391c3740 4 client.25477 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:12:11.255 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.258+0000 7fe8391c3740 1 -- 172.21.15.106:0/512335521 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558242fd33a0 con 0x5582430e1160 2024-03-28T02:12:11.255 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.258+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5582430e1160 0x5582430e3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:11.255 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.258+0000 7fe80f7fe640 10 client.25477 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:11.255 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.258+0000 7fe82f7fe640 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5582430e1160 0x5582430e3550 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1aaecf7a2ab405ef server_cookie=e63f55cbf739ef48 in_seq=0 out_seq=0 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe80f7fe640 1 -- 172.21.15.106:0/512335521 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558242fd33a0 con 0x5582430e1160 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe80f7fe640 10 client.25477 handle_command_reply: tid=0 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe8391c3740 1 client.25477 shutdown 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe8391c3740 1 -- 172.21.15.106:0/512335521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 msgr2=0x55824306a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe8391c3740 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x55824306a2d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe82002c820 tx=0x7fe8200bc500 comp rx=0 tx=0).stop 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe8391c3740 1 -- 172.21.15.106:0/512335521 shutdown_connections 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe8391c3740 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5582430e1160 0x5582430e3550 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe8391c3740 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558243070e00 0x55824306ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe8391c3740 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558243068030 0x55824306a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe8391c3740 1 --2- 172.21.15.106:0/512335521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558243067710 0x55824306a2d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.257 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8391c3740 1 -- 172.21.15.106:0/512335521 >> 172.21.15.106:0/512335521 conn(0x558243062030 msgr2=0x55824306cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:11.258 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.263+0000 7fe80f7fe640 0 client.25477 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:11.258 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8391c3740 1 -- 172.21.15.106:0/512335521 shutdown_connections 2024-03-28T02:12:11.258 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8391c3740 1 -- 172.21.15.106:0/512335521 wait complete. 2024-03-28T02:12:11.258 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8391c3740 20 client.25477 trim_cache size 0 max 16384 2024-03-28T02:12:11.258 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:12:11.258 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:12:11.258 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:12:11.258 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:12:11.259 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 -- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8180a2b30 msgr2=0x7fe8180a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:11.259 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8180a2b30 0x7fe8180a4ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe828005c40 tx=0x7fe828004ab0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.259 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 -- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe830100cb0 msgr2=0x7fe830198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:11.259 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe830100cb0 0x7fe830198c50 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe820009230 tx=0x7fe82002cb20 comp rx=0 tx=0).stop 2024-03-28T02:12:11.259 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 -- 172.21.15.106:0/1911468643 shutdown_connections 2024-03-28T02:12:11.259 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe8180a2b30 0x7fe8180a4ff0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.259 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8301065d0 0x7fe8301a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.260 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe830101670 0x7fe830199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.260 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 --2- 172.21.15.106:0/1911468643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe830100cb0 0x7fe830198c50 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:11.260 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.264+0000 7fe8364f8640 1 -- 172.21.15.106:0/1911468643 >> 172.21.15.106:0/1911468643 conn(0x7fe830076b30 msgr2=0x7fe8300fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:11.260 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.265+0000 7fe8364f8640 1 -- 172.21.15.106:0/1911468643 shutdown_connections 2024-03-28T02:12:11.260 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:11.265+0000 7fe8364f8640 1 -- 172.21.15.106:0/1911468643 wait complete. 2024-03-28T02:12:11.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:11 smithi106 ceph-mon[31965]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-28T02:12:11.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:11 smithi178 ceph-mon[36521]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-28T02:12:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:11 smithi153 ceph-mon[36228]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-28T02:12:13.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:13 smithi106 ceph-mon[31965]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-28T02:12:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:13 smithi178 ceph-mon[36521]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-28T02:12:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:13 smithi153 ceph-mon[36228]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-28T02:12:15.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:15 smithi106 ceph-mon[31965]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-28T02:12:15.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:15 smithi178 ceph-mon[36521]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-28T02:12:15.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:15 smithi153 ceph-mon[36228]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-28T02:12:17.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:17 smithi106 ceph-mon[31965]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-28T02:12:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:17 smithi178 ceph-mon[36521]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-28T02:12:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:17 smithi153 ceph-mon[36228]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-28T02:12:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:19 smithi106 ceph-mon[31965]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-28T02:12:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:19 smithi178 ceph-mon[36521]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-28T02:12:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:19 smithi153 ceph-mon[36228]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-28T02:12:21.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:21 smithi106 ceph-mon[31965]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-28T02:12:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:21 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:12:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:21 smithi178 ceph-mon[36521]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-28T02:12:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:21 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:12:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:21 smithi153 ceph-mon[36228]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-28T02:12:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:21 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:12:23.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:22 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:22 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:22 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:22 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:22 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:22 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:22 smithi106 ceph-mon[31965]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-28T02:12:23.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:22 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:22 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:22 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:22 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:22 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:22 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:22 smithi178 ceph-mon[36521]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-28T02:12:23.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:22 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:22 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:22 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:22 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:22 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:22 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:23.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:22 smithi153 ceph-mon[36228]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-28T02:12:24.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:23 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:12:24.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:23 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:12:24.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:23 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:23 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:12:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:23 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:12:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:23 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:24.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:23 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:12:24.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:23 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:12:24.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:23 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:12:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.397+0000 7fbe2431b640 1 -- 172.21.15.106:0/2866809501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe1c1007f0 msgr2=0x7fbe1c100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.397+0000 7fbe2431b640 1 --2- 172.21.15.106:0/2866809501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe1c1007f0 0x7fbe1c100c70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009930 tx=0x7fbe0c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:12:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.397+0000 7fbe2431b640 1 -- 172.21.15.106:0/2866809501 shutdown_connections 2024-03-28T02:12:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.397+0000 7fbe2431b640 1 --2- 172.21.15.106:0/2866809501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe1c1011b0 0x7fbe1c10d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.397+0000 7fbe2431b640 1 --2- 172.21.15.106:0/2866809501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe1c1007f0 0x7fbe1c100c70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.397+0000 7fbe2431b640 1 --2- 172.21.15.106:0/2866809501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe1c105540 0x7fbe1c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.397+0000 7fbe2431b640 1 -- 172.21.15.106:0/2866809501 >> 172.21.15.106:0/2866809501 conn(0x7fbe1c0773f0 msgr2=0x7fbe1c077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.398+0000 7fbe2431b640 1 -- 172.21.15.106:0/2866809501 shutdown_connections 2024-03-28T02:12:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.398+0000 7fbe2431b640 1 -- 172.21.15.106:0/2866809501 wait complete. 2024-03-28T02:12:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.398+0000 7fbe2431b640 1 Processor -- start 2024-03-28T02:12:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.398+0000 7fbe2431b640 1 -- start start 2024-03-28T02:12:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe2431b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe1c1007f0 0x7fbe1c195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe2431b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe1c1011b0 0x7fbe1c1962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe2431b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe1c105540 0x7fbe1c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe2431b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c10f9a0 con 0x7fbe1c1011b0 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe2431b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c10f820 con 0x7fbe1c1007f0 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe2431b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c10fb20 con 0x7fbe1c105540 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe218b6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe1c1011b0 0x7fbe1c1962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe220b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe1c1007f0 0x7fbe1c195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe218b6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe1c1011b0 0x7fbe1c1962e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49822/0 (socket says 172.21.15.106:49822) 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe228b8640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe1c105540 0x7fbe1c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe218b6640 1 -- 172.21.15.106:0/3023810458 learned_addr learned my addr 172.21.15.106:0/3023810458 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe220b7640 1 -- 172.21.15.106:0/3023810458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe1c105540 msgr2=0x7fbe1c19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe220b7640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe1c105540 0x7fbe1c19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe220b7640 1 -- 172.21.15.106:0/3023810458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe1c1011b0 msgr2=0x7fbe1c1962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:24.395 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe220b7640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe1c1011b0 0x7fbe1c1962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.395 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe220b7640 1 -- 172.21.15.106:0/3023810458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c19ad40 con 0x7fbe1c1007f0 2024-03-28T02:12:24.395 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe228b8640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe1c105540 0x7fbe1c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:24.395 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe220b7640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe1c1007f0 0x7fbe1c195da0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800c380 tx=0x7fbe18016f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:24.395 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe137fe640 1 -- 172.21.15.106:0/3023810458 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe18015e20 con 0x7fbe1c1007f0 2024-03-28T02:12:24.395 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.399+0000 7fbe137fe640 1 -- 172.21.15.106:0/3023810458 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe18005500 con 0x7fbe1c1007f0 2024-03-28T02:12:24.395 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.400+0000 7fbe137fe640 1 -- 172.21.15.106:0/3023810458 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1801e700 con 0x7fbe1c1007f0 2024-03-28T02:12:24.395 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.400+0000 7fbe2431b640 1 -- 172.21.15.106:0/3023810458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1c19b030 con 0x7fbe1c1007f0 2024-03-28T02:12:24.395 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.400+0000 7fbe2431b640 1 -- 172.21.15.106:0/3023810458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1c1a28c0 con 0x7fbe1c1007f0 2024-03-28T02:12:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.403+0000 7fbe137fe640 1 -- 172.21.15.106:0/3023810458 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 24) v1 ==== 137524+0+0 (secure 0 0 0) 0x7fbe18025020 con 0x7fbe1c1007f0 2024-03-28T02:12:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.404+0000 7fbe26fe4740 1 Processor -- start 2024-03-28T02:12:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.404+0000 7fbe26fe4740 1 -- start start 2024-03-28T02:12:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.404+0000 7fbe137fe640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbe080a2a20 0x7fbe080a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.404+0000 7fbe137fe640 1 -- 172.21.15.106:0/3023810458 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fbe180d72b0 con 0x7fbe1c1007f0 2024-03-28T02:12:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.404+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd266af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.404+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.404+0000 7fbe228b8640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbe080a2a20 0x7fbe080a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd26fe00 0x55f8cd2721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd1cab10 con 0x55f8cd266710 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd173900 con 0x55f8cd267030 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd1c6bb0 con 0x55f8cd26fe00 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe218b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe218b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:37672/0 (socket says 172.21.15.106:37672) 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe218b6640 1 -- 172.21.15.106:0/1754792920 learned_addr learned my addr 172.21.15.106:0/1754792920 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe220b7640 1 --2- 172.21.15.106:0/1754792920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd266af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe218b6640 1 -- 172.21.15.106:0/1754792920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd26fe00 msgr2=0x55f8cd2721f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe218b6640 1 --2- 172.21.15.106:0/1754792920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd26fe00 0x55f8cd2721f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe218b6640 1 -- 172.21.15.106:0/1754792920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 msgr2=0x55f8cd266af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe218b6640 1 --2- 172.21.15.106:0/1754792920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd266af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe218b6640 1 -- 172.21.15.106:0/1754792920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8ccedf210 con 0x55f8cd267030 2024-03-28T02:12:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe218b6640 1 --2- 172.21.15.106:0/1754792920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26f8c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c036040 tx=0x7fbe0c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=392de94585a4bc8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe228b8640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbe080a2a20 0x7fbe080a4ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe14005fd0 tx=0x7fbe1400d040 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe10bfe640 1 -- 172.21.15.106:0/1754792920 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c045070 con 0x55f8cd267030 2024-03-28T02:12:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe10bfe640 1 -- 172.21.15.106:0/1754792920 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe0c02c880 con 0x55f8cd267030 2024-03-28T02:12:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.405+0000 7fbe10bfe640 1 -- 172.21.15.106:0/1754792920 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c004b60 con 0x55f8cd267030 2024-03-28T02:12:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.406+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1754792920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 msgr2=0x55f8cd26f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.406+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1754792920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26f8c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c036040 tx=0x7fbe0c009560 comp rx=0 tx=0).stop 2024-03-28T02:12:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.406+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1754792920 shutdown_connections 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.406+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1754792920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd26fe00 0x55f8cd2721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.406+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1754792920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26f8c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.406+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1754792920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd266af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.407+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1754792920 >> 172.21.15.106:0/1754792920 conn(0x55f8cd261030 msgr2=0x55f8cd261410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.407+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1754792920 shutdown_connections 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.407+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1754792920 wait complete. 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.407+0000 7fbe26fe4740 1 -- start start 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.407+0000 7fbe26fe4740 1 Processor -- start 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd2729d0 con 0x55f8cd26fe00 2024-03-28T02:12:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd186d50 con 0x55f8cd267030 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd272b50 con 0x55f8cd266710 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe26fe4740 10 client.0 _ll_register_callbacks cb 0x55f8cd186720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe228b8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe228b8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49838/0 (socket says 172.21.15.106:49838) 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe228b8640 1 -- 172.21.15.106:0/3082384651 learned_addr learned my addr 172.21.15.106:0/3082384651 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe220b7640 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe218b6640 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe220b7640 1 -- 172.21.15.106:0/3082384651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 msgr2=0x55f8cd26d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe220b7640 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe220b7640 1 -- 172.21.15.106:0/3082384651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 msgr2=0x55f8cd26d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe220b7640 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe220b7640 1 -- 172.21.15.106:0/3082384651 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8ccfab620 con 0x55f8cd266710 2024-03-28T02:12:24.406 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe228b8640 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe218b6640 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe220b7640 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26ce40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbe180d44a0 tx=0x7fbe180a33a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff6e3ea03fee140a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.178:3300/0 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.408+0000 7fbe05ffb640 1 -- 172.21.15.106:0/3082384651 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe180a3740 con 0x55f8cd266710 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe26fe4740 1 -- 172.21.15.106:0/3082384651 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe0c02c880 con 0x55f8cd266710 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe05ffb640 1 -- 172.21.15.106:0/3082384651 <== mon.1 v2:172.21.15.178:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fbe180ab5f0 con 0x55f8cd266710 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe05ffb640 1 -- 172.21.15.106:0/3082384651 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe180ab7f0 con 0x55f8cd266710 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe26fe4740 10 client.25486 fetch_fsmap learned FSMap version 29 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe26fe4740 1 -- 172.21.15.106:0/3082384651 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f8cd27ed40 con 0x55f8cd266710 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe05ffb640 1 -- 172.21.15.106:0/3082384651 <== mon.1 v2:172.21.15.178:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe0c02c880 con 0x55f8cd266710 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe05ffb640 1 -- 172.21.15.106:0/3082384651 <== mon.1 v2:172.21.15.178:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fbe180a2730 con 0x55f8cd266710 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe26fe4740 10 client.25486 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:12:24.407 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe26fe4740 10 client.25486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe26fe4740 20 client.25486 populate_metadata read hostname 'smithi106' 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f8cd2801b0 0x55f8cd2825a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe26fe4740 4 client.25486 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe26fe4740 1 -- 172.21.15.106:0/3082384651 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8cd1d8050 con 0x55f8cd2801b0 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.409+0000 7fbe218b6640 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f8cd2801b0 0x55f8cd2825a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.410+0000 7fbe05ffb640 10 client.25486 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.410+0000 7fbe218b6640 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f8cd2801b0 0x55f8cd2825a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=beb062f861a4961f server_cookie=5014e66957f3efbd in_seq=0 out_seq=0 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.410+0000 7fbe05ffb640 1 -- 172.21.15.106:0/3082384651 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f8cd1d8050 con 0x55f8cd2801b0 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.410+0000 7fbe05ffb640 10 client.25486 handle_command_reply: tid=0 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.410+0000 7fbe26fe4740 1 client.25486 shutdown 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.410+0000 7fbe26fe4740 1 -- 172.21.15.106:0/3082384651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 msgr2=0x55f8cd26ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.410+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26ce40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbe180d44a0 tx=0x7fbe180a33a0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe26fe4740 1 -- 172.21.15.106:0/3082384651 shutdown_connections 2024-03-28T02:12:24.408 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f8cd2801b0 0x55f8cd2825a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe05ffb640 0 client.25486 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/3082384651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26ce40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe26fe4740 1 -- 172.21.15.106:0/3082384651 >> 172.21.15.106:0/3082384651 conn(0x55f8cd261030 msgr2=0x55f8cd262d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe26fe4740 1 -- 172.21.15.106:0/3082384651 shutdown_connections 2024-03-28T02:12:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe26fe4740 1 -- 172.21.15.106:0/3082384651 wait complete. 2024-03-28T02:12:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.411+0000 7fbe26fe4740 20 client.25486 trim_cache size 0 max 16384 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.424+0000 7fbe26fe4740 1 Processor -- start 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe26fe4740 1 -- start start 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c1ac230 con 0x55f8cd26fe00 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c1adbb0 con 0x55f8cd267030 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c1ac890 con 0x55f8cd266710 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe220b7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe218b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe228b8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe218b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:37686/0 (socket says 172.21.15.106:37686) 2024-03-28T02:12:24.419 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe220b7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:46908/0 (socket says 172.21.15.106:46908) 2024-03-28T02:12:24.419 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe218b6640 1 -- 172.21.15.106:0/2750102323 learned_addr learned my addr 172.21.15.106:0/2750102323 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe228b8640 1 -- 172.21.15.106:0/2750102323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 msgr2=0x55f8cd26f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe228b8640 1 --2- 172.21.15.106:0/2750102323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe228b8640 1 -- 172.21.15.106:0/2750102323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 msgr2=0x55f8cd26b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe228b8640 1 --2- 172.21.15.106:0/2750102323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe228b8640 1 -- 172.21.15.106:0/2750102323 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8cd266e60 con 0x55f8cd26fe00 2024-03-28T02:12:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe218b6640 1 --2- 172.21.15.106:0/2750102323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe220b7640 1 --2- 172.21.15.106:0/2750102323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.425+0000 7fbe228b8640 1 --2- 172.21.15.106:0/2750102323 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26bc50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400b480 tx=0x7fbe14014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff155be02a34482f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.426+0000 7fbe07fff640 1 -- 172.21.15.106:0/2750102323 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe14016930 con 0x55f8cd26fe00 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.426+0000 7fbe07fff640 1 -- 172.21.15.106:0/2750102323 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe14029070 con 0x55f8cd26fe00 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.426+0000 7fbe07fff640 1 -- 172.21.15.106:0/2750102323 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe14024cd0 con 0x55f8cd26fe00 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 -- 172.21.15.106:0/2750102323 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 msgr2=0x55f8cd26bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/2750102323 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26bc50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400b480 tx=0x7fbe14014000 comp rx=0 tx=0).stop 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 -- 172.21.15.106:0/2750102323 shutdown_connections 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/2750102323 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd26fe00 0x55f8cd26bc50 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/2750102323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd267030 0x55f8cd26b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/2750102323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd266710 0x55f8cd26f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 -- 172.21.15.106:0/2750102323 >> 172.21.15.106:0/2750102323 conn(0x55f8cd261030 msgr2=0x55f8cd267590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 -- 172.21.15.106:0/2750102323 shutdown_connections 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 -- 172.21.15.106:0/2750102323 wait complete. 2024-03-28T02:12:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 -- start start 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 Processor -- start 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd272f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd267030 0x55f8cd273490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.427+0000 7fbe26fe4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd26fe00 0x55f8cd2d9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c1ad550 con 0x55f8cd266710 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c1acef0 con 0x55f8cd26fe00 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe26fe4740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c1aa3f0 con 0x55f8cd267030 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe26fe4740 10 client.0 _ll_register_callbacks cb 0x7fbe180a39e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe220b7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd272f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe218b6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd267030 0x55f8cd273490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe220b7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd272f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49858/0 (socket says 172.21.15.106:49858) 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe220b7640 1 -- 172.21.15.106:0/1085863309 learned_addr learned my addr 172.21.15.106:0/1085863309 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe218b6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd267030 0x55f8cd273490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:46912/0 (socket says 172.21.15.106:46912) 2024-03-28T02:12:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe218b6640 1 -- 172.21.15.106:0/1085863309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd26fe00 msgr2=0x55f8cd2d9a10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe218b6640 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd26fe00 0x55f8cd2d9a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe218b6640 1 -- 172.21.15.106:0/1085863309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 msgr2=0x55f8cd272f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe218b6640 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd272f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe218b6640 1 -- 172.21.15.106:0/1085863309 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8cd2da020 con 0x55f8cd267030 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe220b7640 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd272f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe218b6640 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd267030 0x55f8cd273490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009850 tx=0x7fbe0c009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=148c8f5bddfddb53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.178:3300/0 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe05ffb640 1 -- 172.21.15.106:0/1085863309 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c053070 con 0x55f8cd267030 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1085863309 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe14029070 con 0x55f8cd267030 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.428+0000 7fbe05ffb640 1 -- 172.21.15.106:0/1085863309 <== mon.1 v2:172.21.15.178:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fbe0c004040 con 0x55f8cd267030 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe05ffb640 1 -- 172.21.15.106:0/1085863309 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe0c05f050 con 0x55f8cd267030 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe26fe4740 10 client.25492 fetch_fsmap learned FSMap version 29 2024-03-28T02:12:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1085863309 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f8cd2db320 con 0x55f8cd267030 2024-03-28T02:12:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe05ffb640 1 -- 172.21.15.106:0/1085863309 <== mon.1 v2:172.21.15.178:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe14029070 con 0x55f8cd267030 2024-03-28T02:12:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe05ffb640 1 -- 172.21.15.106:0/1085863309 <== mon.1 v2:172.21.15.178:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fbe0c0035f0 con 0x55f8cd267030 2024-03-28T02:12:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe26fe4740 10 client.25492 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:12:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe26fe4740 10 client.25492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:12:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe26fe4740 20 client.25492 populate_metadata read hostname 'smithi106' 2024-03-28T02:12:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f8cd2801b0 0x55f8cd2dd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe26fe4740 4 client.25492 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:12:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1085863309 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f8cd1d23a0 con 0x55f8cd2801b0 2024-03-28T02:12:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe220b7640 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f8cd2801b0 0x55f8cd2dd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe05ffb640 10 client.25492 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.429+0000 7fbe220b7640 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f8cd2801b0 0x55f8cd2dd670 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=703b258eb58b71 server_cookie=2fe2a1f45063a298 in_seq=0 out_seq=0 2024-03-28T02:12:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.432+0000 7fbe05ffb640 1 -- 172.21.15.106:0/1085863309 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f8cd1d23a0 con 0x55f8cd2801b0 2024-03-28T02:12:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.432+0000 7fbe05ffb640 10 client.25492 handle_command_reply: tid=0 2024-03-28T02:12:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.432+0000 7fbe26fe4740 1 client.25492 shutdown 2024-03-28T02:12:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1085863309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd267030 msgr2=0x55f8cd273490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd267030 0x55f8cd273490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009850 tx=0x7fbe0c009680 comp rx=0 tx=0).stop 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1085863309 shutdown_connections 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f8cd2801b0 0x55f8cd2dd670 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8cd26fe00 0x55f8cd2d9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe05ffb640 0 client.25492 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f8cd267030 0x55f8cd273490 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 --2- 172.21.15.106:0/1085863309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8cd266710 0x55f8cd272f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1085863309 >> 172.21.15.106:0/1085863309 conn(0x55f8cd261030 msgr2=0x55f8cd26b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1085863309 shutdown_connections 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 1 -- 172.21.15.106:0/1085863309 wait complete. 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.433+0000 7fbe26fe4740 20 client.25492 trim_cache size 0 max 16384 2024-03-28T02:12:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:12:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:12:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:12:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:12:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 -- 172.21.15.106:0/3023810458 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbe080a2a20 msgr2=0x7fbe080a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbe080a2a20 0x7fbe080a4ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe14005fd0 tx=0x7fbe1400d040 comp rx=0 tx=0).stop 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 -- 172.21.15.106:0/3023810458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe1c1007f0 msgr2=0x7fbe1c195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe1c1007f0 0x7fbe1c195da0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800c380 tx=0x7fbe18016f90 comp rx=0 tx=0).stop 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 -- 172.21.15.106:0/3023810458 shutdown_connections 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbe080a2a20 0x7fbe080a4ee0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe1c105540 0x7fbe1c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe1c1011b0 0x7fbe1c1962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 --2- 172.21.15.106:0/3023810458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe1c1007f0 0x7fbe1c195da0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 -- 172.21.15.106:0/3023810458 >> 172.21.15.106:0/3023810458 conn(0x7fbe1c0773f0 msgr2=0x7fbe1c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 -- 172.21.15.106:0/3023810458 shutdown_connections 2024-03-28T02:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:24.434+0000 7fbe2431b640 1 -- 172.21.15.106:0/3023810458 wait complete. 2024-03-28T02:12:25.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:24 smithi106 ceph-mon[31965]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:12:25.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:24 smithi178 ceph-mon[36521]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:12:25.147 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:24 smithi153 ceph-mon[36228]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:12:27.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:27 smithi153 ceph-mon[36228]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:12:27.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:27 smithi106 ceph-mon[31965]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:12:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:27 smithi178 ceph-mon[36521]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:12:29.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:29 smithi153 ceph-mon[36228]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:29.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:29 smithi106 ceph-mon[31965]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:29 smithi178 ceph-mon[36521]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:31.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:31 smithi153 ceph-mon[36228]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:31.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:31 smithi106 ceph-mon[31965]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:31 smithi178 ceph-mon[36521]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:33 smithi153 ceph-mon[36228]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:33.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:33 smithi106 ceph-mon[31965]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:33 smithi178 ceph-mon[36521]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:35.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:35 smithi106 ceph-mon[31965]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:35.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:35 smithi153 ceph-mon[36228]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:35 smithi178 ceph-mon[36521]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.563+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/805452862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c08105540 msgr2=0x7f1c08105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.563+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/805452862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c08105540 0x7f1c08105920 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009900 tx=0x7f1bfc02bc30 comp rx=0 tx=0).stop 2024-03-28T02:12:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.563+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/805452862 shutdown_connections 2024-03-28T02:12:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.563+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/805452862 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c081011b0 0x7f1c0810d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.563+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/805452862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c081007f0 0x7f1c08100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.563+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/805452862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c08105540 0x7f1c08105920 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.563+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/805452862 >> 172.21.15.106:0/805452862 conn(0x7f1c080773f0 msgr2=0x7f1c08077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.563+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/805452862 shutdown_connections 2024-03-28T02:12:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.563+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/805452862 wait complete. 2024-03-28T02:12:37.558 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.564+0000 7f1c0ecd1640 1 Processor -- start 2024-03-28T02:12:37.558 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.564+0000 7f1c0ecd1640 1 -- start start 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.564+0000 7f1c0ecd1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c081007f0 0x7f1c08198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.564+0000 7f1c0ecd1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c081011b0 0x7f1c08199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.564+0000 7f1c0ecd1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c08105540 0x7f1c081a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c0ecd1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08110a60 con 0x7f1c081007f0 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c0ca6d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c081007f0 0x7f1c08198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c0ca6d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c081007f0 0x7f1c08198de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:44424/0 (socket says 172.21.15.106:44424) 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c0ca6d640 1 -- 172.21.15.106:0/3706443233 learned_addr learned my addr 172.21.15.106:0/3706443233 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c0d26e640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c08105540 0x7f1c081a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081108e0 con 0x7f1c081011b0 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08110be0 con 0x7f1c08105540 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c07fff640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c081011b0 0x7f1c08199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c07fff640 1 -- 172.21.15.106:0/3706443233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c08105540 msgr2=0x7f1c081a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c07fff640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c08105540 0x7f1c081a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c07fff640 1 -- 172.21.15.106:0/3706443233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c081007f0 msgr2=0x7f1c08198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c07fff640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c081007f0 0x7f1c08198de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c07fff640 1 -- 172.21.15.106:0/3706443233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c081a43a0 con 0x7f1c081011b0 2024-03-28T02:12:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c0d26e640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c08105540 0x7f1c081a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c07fff640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c081011b0 0x7f1c08199320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400e9f0 tx=0x7f1bf400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.565+0000 7f1c05ffb640 1 -- 172.21.15.106:0/3706443233 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf400ccc0 con 0x7f1c081011b0 2024-03-28T02:12:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.566+0000 7f1c05ffb640 1 -- 172.21.15.106:0/3706443233 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bf4010910 con 0x7f1c081011b0 2024-03-28T02:12:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.566+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c081a4570 con 0x7f1c081011b0 2024-03-28T02:12:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.566+0000 7f1c05ffb640 1 -- 172.21.15.106:0/3706443233 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4018960 con 0x7f1c081011b0 2024-03-28T02:12:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.566+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c081a4a10 con 0x7f1c081011b0 2024-03-28T02:12:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.569+0000 7f1c11997740 1 Processor -- start 2024-03-28T02:12:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.569+0000 7f1c11997740 1 -- start start 2024-03-28T02:12:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.569+0000 7f1c05ffb640 1 -- 172.21.15.106:0/3706443233 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 24) v1 ==== 137524+0+0 (secure 0 0 0) 0x7f1bf4010430 con 0x7f1c081011b0 2024-03-28T02:12:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.569+0000 7f1c11997740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f08710 0x563cb1f08af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.569+0000 7f1c11997740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f09030 0x563cb1f118c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.569+0000 7f1c11997740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f11e00 0x563cb1f141f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.569+0000 7f1c11997740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb1e6cb10 con 0x563cb1f11e00 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.569+0000 7f1c11997740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb1e15900 con 0x563cb1f08710 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.569+0000 7f1c11997740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb1e68bb0 con 0x563cb1f09030 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0ca6d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f09030 0x563cb1f118c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0ca6d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f09030 0x563cb1f118c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:37256/0 (socket says 172.21.15.106:37256) 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0ca6d640 1 -- 172.21.15.106:0/1180309541 learned_addr learned my addr 172.21.15.106:0/1180309541 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0d26e640 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f08710 0x563cb1f08af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c07fff640 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f11e00 0x563cb1f141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c05ffb640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1bf80a2b30 0x7f1bf80a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0ca6d640 1 -- 172.21.15.106:0/1180309541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f08710 msgr2=0x563cb1f08af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0ca6d640 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f08710 0x563cb1f08af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c05ffb640 1 -- 172.21.15.106:0/3706443233 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f1bf40cf8f0 con 0x7f1c081011b0 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0ca6d640 1 -- 172.21.15.106:0/1180309541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f11e00 msgr2=0x563cb1f141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0ca6d640 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f11e00 0x563cb1f141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0ca6d640 1 -- 172.21.15.106:0/1180309541 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cb1b81210 con 0x563cb1f09030 2024-03-28T02:12:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0d26e640 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f08710 0x563cb1f08af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:12:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c0d26e640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1bf80a2b30 0x7f1bf80a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.570+0000 7f1c07fff640 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f11e00 0x563cb1f141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:12:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.571+0000 7f1c0ca6d640 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f09030 0x563cb1f118c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc036040 tx=0x7f1bfc009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d54726d68f7468c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.571+0000 7f1c0d26e640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1bf80a2b30 0x7f1bf80a4ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1c000094d0 tx=0x7f1c00014040 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.571+0000 7f1bf2ffd640 1 -- 172.21.15.106:0/1180309541 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc043070 con 0x563cb1f09030 2024-03-28T02:12:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.571+0000 7f1bf2ffd640 1 -- 172.21.15.106:0/1180309541 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bfc02c5d0 con 0x563cb1f09030 2024-03-28T02:12:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.571+0000 7f1bf2ffd640 1 -- 172.21.15.106:0/1180309541 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc043070 con 0x563cb1f09030 2024-03-28T02:12:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 -- 172.21.15.106:0/1180309541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f09030 msgr2=0x563cb1f118c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f09030 0x563cb1f118c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc036040 tx=0x7f1bfc009560 comp rx=0 tx=0).stop 2024-03-28T02:12:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 -- 172.21.15.106:0/1180309541 shutdown_connections 2024-03-28T02:12:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f11e00 0x563cb1f141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f09030 0x563cb1f118c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 --2- 172.21.15.106:0/1180309541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f08710 0x563cb1f08af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 -- 172.21.15.106:0/1180309541 >> 172.21.15.106:0/1180309541 conn(0x563cb1f03030 msgr2=0x563cb1f03410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 -- 172.21.15.106:0/1180309541 shutdown_connections 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 -- 172.21.15.106:0/1180309541 wait complete. 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 -- start start 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.572+0000 7f1c11997740 1 Processor -- start 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c11997740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f0eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c11997740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f0f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c11997740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 0x563cb1f22250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c11997740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb1f18de0 con 0x563cb1f08710 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c11997740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb1e28d50 con 0x563cb1f11e00 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c11997740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb1f18f60 con 0x563cb1f0f930 2024-03-28T02:12:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c11997740 10 client.0 _ll_register_callbacks cb 0x563cb1e28720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c07fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f0f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c07fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f0f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:60588/0 (socket says 172.21.15.106:60588) 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c07fff640 1 -- 172.21.15.106:0/2661054732 learned_addr learned my addr 172.21.15.106:0/2661054732 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c0d26e640 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 0x563cb1f22250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c0ca6d640 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f0eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c07fff640 1 -- 172.21.15.106:0/2661054732 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 msgr2=0x563cb1f22250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c07fff640 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 0x563cb1f22250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c07fff640 1 -- 172.21.15.106:0/2661054732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 msgr2=0x563cb1f0eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c07fff640 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f0eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.573+0000 7f1c07fff640 1 -- 172.21.15.106:0/2661054732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cb1c4d620 con 0x563cb1f11e00 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.574+0000 7f1c07fff640 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f0f3f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4097de0 tx=0x7f1bf409b4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acff526728fb456a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.574+0000 7f1be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-28T02:12:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.574+0000 7f1be7fff640 1 -- 172.21.15.106:0/2661054732 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf409c080 con 0x563cb1f11e00 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.574+0000 7f1be7fff640 1 -- 172.21.15.106:0/2661054732 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f1bf40cce20 con 0x563cb1f11e00 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.574+0000 7f1c11997740 1 -- 172.21.15.106:0/2661054732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1bfc02c5d0 con 0x563cb1f11e00 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.574+0000 7f1be7fff640 1 -- 172.21.15.106:0/2661054732 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bf409b980 con 0x563cb1f11e00 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.574+0000 7f1c11997740 10 client.25583 fetch_fsmap learned FSMap version 29 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.574+0000 7f1c11997740 1 -- 172.21.15.106:0/2661054732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563cb1f25510 con 0x563cb1f11e00 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.574+0000 7f1be7fff640 1 -- 172.21.15.106:0/2661054732 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bfc02c5d0 con 0x563cb1f11e00 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1be7fff640 1 -- 172.21.15.106:0/2661054732 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f1bf409ae60 con 0x563cb1f11e00 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1c11997740 10 client.25583 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1c11997740 10 client.25583 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1c11997740 20 client.25583 populate_metadata read hostname 'smithi106' 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1c11997740 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x563cb1f26920 0x563cb1f28d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1c11997740 4 client.25583 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1c11997740 1 -- 172.21.15.106:0/2661054732 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cb1e7a050 con 0x563cb1f26920 2024-03-28T02:12:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1c0ca6d640 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x563cb1f26920 0x563cb1f28d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1c0ca6d640 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x563cb1f26920 0x563cb1f28d30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da8cc796b6d76751 server_cookie=d6996b388a3a5472 in_seq=0 out_seq=0 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.575+0000 7f1be7fff640 10 client.25583 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1be7fff640 1 -- 172.21.15.106:0/2661054732 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563cb1e7a050 con 0x563cb1f26920 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1be7fff640 10 client.25583 handle_command_reply: tid=0 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 client.25583 shutdown 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 -- 172.21.15.106:0/2661054732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 msgr2=0x563cb1f0f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f0f3f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4097de0 tx=0x7f1bf409b4c0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 -- 172.21.15.106:0/2661054732 shutdown_connections 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x563cb1f26920 0x563cb1f28d30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 0x563cb1f22250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1be7fff640 0 client.25583 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f0f3f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 --2- 172.21.15.106:0/2661054732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f0eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 -- 172.21.15.106:0/2661054732 >> 172.21.15.106:0/2661054732 conn(0x563cb1f03030 msgr2=0x563cb1f04d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 -- 172.21.15.106:0/2661054732 shutdown_connections 2024-03-28T02:12:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 1 -- 172.21.15.106:0/2661054732 wait complete. 2024-03-28T02:12:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.576+0000 7f1c11997740 20 client.25583 trim_cache size 0 max 16384 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c11997740 1 Processor -- start 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c11997740 1 -- start start 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c11997740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f0d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c11997740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f0f930 0x563cb1f0dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c11997740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f11e00 0x563cb1f0c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c11997740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081ac650 con 0x563cb1f08710 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c11997740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081adfd0 con 0x563cb1f0f930 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c11997740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081accb0 con 0x563cb1f11e00 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c0ca6d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f0d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c07fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f0f930 0x563cb1f0dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c07fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f0f930 0x563cb1f0dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:60590/0 (socket says 172.21.15.106:60590) 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c07fff640 1 -- 172.21.15.106:0/1326813898 learned_addr learned my addr 172.21.15.106:0/1326813898 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.603+0000 7f1c0d26e640 1 --2- 172.21.15.106:0/1326813898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f11e00 0x563cb1f0c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.604+0000 7f1c07fff640 1 -- 172.21.15.106:0/1326813898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f11e00 msgr2=0x563cb1f0c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.604+0000 7f1c07fff640 1 --2- 172.21.15.106:0/1326813898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f11e00 0x563cb1f0c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.604+0000 7f1c07fff640 1 -- 172.21.15.106:0/1326813898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 msgr2=0x563cb1f0d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.604+0000 7f1c07fff640 1 --2- 172.21.15.106:0/1326813898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f0d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.604+0000 7f1c07fff640 1 -- 172.21.15.106:0/1326813898 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cb1f08e60 con 0x563cb1f0f930 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.604+0000 7f1c07fff640 1 --2- 172.21.15.106:0/1326813898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f0f930 0x563cb1f0dd00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4098350 tx=0x7f1bf409b5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc0a9f420a944146 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.604+0000 7f1bf27fc640 1 -- 172.21.15.106:0/1326813898 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf409c080 con 0x563cb1f0f930 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.604+0000 7f1bf27fc640 1 -- 172.21.15.106:0/1326813898 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bf4010720 con 0x563cb1f0f930 2024-03-28T02:12:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.604+0000 7f1bf27fc640 1 -- 172.21.15.106:0/1326813898 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40cce20 con 0x563cb1f0f930 2024-03-28T02:12:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 -- 172.21.15.106:0/1326813898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f0f930 msgr2=0x563cb1f0dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 --2- 172.21.15.106:0/1326813898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f0f930 0x563cb1f0dd00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4098350 tx=0x7f1bf409b5f0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 -- 172.21.15.106:0/1326813898 shutdown_connections 2024-03-28T02:12:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 --2- 172.21.15.106:0/1326813898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f11e00 0x563cb1f0c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 --2- 172.21.15.106:0/1326813898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f0f930 0x563cb1f0dd00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 --2- 172.21.15.106:0/1326813898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f0d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 -- 172.21.15.106:0/1326813898 >> 172.21.15.106:0/1326813898 conn(0x563cb1f03030 msgr2=0x563cb1f03410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 -- 172.21.15.106:0/1326813898 shutdown_connections 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 -- 172.21.15.106:0/1326813898 wait complete. 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.605+0000 7f1c11997740 1 -- start start 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.606+0000 7f1c11997740 1 Processor -- start 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.606+0000 7f1c11997740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f1b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.606+0000 7f1c11997740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 0x563cb1f1bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.606+0000 7f1c11997740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f1c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.606+0000 7f1c0ca6d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f1b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.606+0000 7f1c0ca6d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f1b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:44466/0 (socket says 172.21.15.106:44466) 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.606+0000 7f1c0ca6d640 1 -- 172.21.15.106:0/3391906207 learned_addr learned my addr 172.21.15.106:0/3391906207 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.606+0000 7f1c07fff640 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 0x563cb1f1bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.606+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081ad970 con 0x563cb1f08710 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081ad310 con 0x563cb1f11e00 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c0d26e640 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f1c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081aa810 con 0x563cb1f0f930 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c11997740 10 client.0 _ll_register_callbacks cb 0x7f1bf4097c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c07fff640 1 -- 172.21.15.106:0/3391906207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 msgr2=0x563cb1f1c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c07fff640 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f1c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c07fff640 1 -- 172.21.15.106:0/3391906207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 msgr2=0x563cb1f1b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c07fff640 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f1b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c07fff640 1 -- 172.21.15.106:0/3391906207 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cb1f81290 con 0x563cb1f0f930 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c07fff640 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 0x563cb1f1bcc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4098350 tx=0x7f1bf409b4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3c700e42cfda317 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.178:3300/0 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1be7fff640 1 -- 172.21.15.106:0/3391906207 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4097a40 con 0x563cb1f0f930 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1be7fff640 1 -- 172.21.15.106:0/3391906207 <== mon.1 v2:172.21.15.178:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f1bf4018b00 con 0x563cb1f0f930 2024-03-28T02:12:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1bf4010720 con 0x563cb1f0f930 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1be7fff640 1 -- 172.21.15.106:0/3391906207 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bf40a4120 con 0x563cb1f0f930 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c11997740 10 client.25510 fetch_fsmap learned FSMap version 29 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.607+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563cb1f81290 con 0x563cb1f0f930 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.608+0000 7f1be7fff640 1 -- 172.21.15.106:0/3391906207 <== mon.1 v2:172.21.15.178:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bf4010720 con 0x563cb1f0f930 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.608+0000 7f1be7fff640 1 -- 172.21.15.106:0/3391906207 <== mon.1 v2:172.21.15.178:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f1bf40a09f0 con 0x563cb1f0f930 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.608+0000 7f1c11997740 10 client.25510 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.608+0000 7f1c11997740 10 client.25510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.608+0000 7f1c11997740 20 client.25510 populate_metadata read hostname 'smithi106' 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.608+0000 7f1c11997740 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x563cb1f26920 0x563cb1f84730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.608+0000 7f1c11997740 4 client.25510 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.608+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563cb1e743a0 con 0x563cb1f26920 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.608+0000 7f1c0ca6d640 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x563cb1f26920 0x563cb1f84730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.609+0000 7f1be7fff640 10 client.25510 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.609+0000 7f1c0ca6d640 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x563cb1f26920 0x563cb1f84730 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1539dc20be8a329 server_cookie=9736baad375b254e in_seq=0 out_seq=0 2024-03-28T02:12:37.605 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.611+0000 7f1be7fff640 1 -- 172.21.15.106:0/3391906207 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563cb1e743a0 con 0x563cb1f26920 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.611+0000 7f1be7fff640 10 client.25510 handle_command_reply: tid=0 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.611+0000 7f1c11997740 1 client.25510 shutdown 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 msgr2=0x563cb1f1bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 0x563cb1f1bcc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4098350 tx=0x7f1bf409b4c0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 shutdown_connections 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x563cb1f26920 0x563cb1f84730 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563cb1f11e00 0x563cb1f1c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563cb1f0f930 0x563cb1f1bcc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 --2- 172.21.15.106:0/3391906207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cb1f08710 0x563cb1f1b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 >> 172.21.15.106:0/3391906207 conn(0x563cb1f03030 msgr2=0x563cb1f01790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1be7fff640 0 client.25510 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 shutdown_connections 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.612+0000 7f1c11997740 1 -- 172.21.15.106:0/3391906207 wait complete. 2024-03-28T02:12:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c11997740 20 client.25510 trim_cache size 0 max 16384 2024-03-28T02:12:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:12:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:12:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:12:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1bf80a2b30 msgr2=0x7f1bf80a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1bf80a2b30 0x7f1bf80a4ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1c000094d0 tx=0x7f1c00014040 comp rx=0 tx=0).stop 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c081011b0 msgr2=0x7f1c08199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c081011b0 0x7f1c08199320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400e9f0 tx=0x7f1bf400eeb0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 shutdown_connections 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1bf80a2b30 0x7f1bf80a4ff0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c08105540 0x7f1c081a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c081011b0 0x7f1c08199320 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 --2- 172.21.15.106:0/3706443233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c081007f0 0x7f1c08198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.613+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 >> 172.21.15.106:0/3706443233 conn(0x7f1c080773f0 msgr2=0x7f1c080ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.614+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 shutdown_connections 2024-03-28T02:12:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:37.614+0000 7f1c0ecd1640 1 -- 172.21.15.106:0/3706443233 wait complete. 2024-03-28T02:12:37.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:37 smithi106 ceph-mon[31965]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:37 smithi178 ceph-mon[36521]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:37.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:37 smithi153 ceph-mon[36228]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:39.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:39 smithi106 ceph-mon[31965]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:39 smithi178 ceph-mon[36521]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:39.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:39 smithi153 ceph-mon[36228]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-03-28T02:12:41.599 INFO:tasks.workunit.client.0.smithi106.stderr: git switch -c 2024-03-28T02:12:41.600 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-03-28T02:12:41.600 INFO:tasks.workunit.client.0.smithi106.stderr:Or undo this operation with: 2024-03-28T02:12:41.600 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-03-28T02:12:41.600 INFO:tasks.workunit.client.0.smithi106.stderr: git switch - 2024-03-28T02:12:41.600 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-03-28T02:12:41.600 INFO:tasks.workunit.client.0.smithi106.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T02:12:41.600 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-03-28T02:12:41.600 INFO:tasks.workunit.client.0.smithi106.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T02:12:41.603 DEBUG:teuthology.orchestra.run.smithi106:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T02:12:41.668 INFO:tasks.workunit.client.0.smithi106.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T02:12:41.670 INFO:tasks.workunit.client.0.smithi106.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T02:12:41.670 INFO:tasks.workunit.client.0.smithi106.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T02:12:41.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:41 smithi106 ceph-mon[31965]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:12:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:41 smithi178 ceph-mon[36521]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:12:41.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:41 smithi153 ceph-mon[36228]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:12:42.429 INFO:tasks.workunit.client.0.smithi106.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T02:12:42.490 INFO:tasks.workunit.client.0.smithi106.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T02:12:42.531 INFO:tasks.workunit.client.0.smithi106.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T02:12:42.533 INFO:tasks.workunit.client.0.smithi106.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T02:12:42.533 INFO:tasks.workunit.client.0.smithi106.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T02:12:42.572 INFO:tasks.workunit.client.0.smithi106.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T02:12:42.575 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:12:42.575 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T02:12:42.631 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-03-28T02:12:42.633 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-03-28T02:12:42.633 DEBUG:teuthology.orchestra.run.smithi106:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-03-28T02:12:42.692 INFO:tasks.workunit.client.0.smithi106.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-28T02:12:42.771 INFO:tasks.workunit.client.0.smithi106.stdout: Iozone: Performance Test of File I/O 2024-03-28T02:12:42.771 INFO:tasks.workunit.client.0.smithi106.stdout: Version $Revision: 3.494 $ 2024-03-28T02:12:42.771 INFO:tasks.workunit.client.0.smithi106.stdout: Compiled for 64 bit mode. 2024-03-28T02:12:42.771 INFO:tasks.workunit.client.0.smithi106.stdout: Build: linux-AMD64 2024-03-28T02:12:42.771 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:12:42.771 INFO:tasks.workunit.client.0.smithi106.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-28T02:12:42.771 INFO:tasks.workunit.client.0.smithi106.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-28T02:12:42.771 INFO:tasks.workunit.client.0.smithi106.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Run began: Thu Mar 28 02:12:42 2024 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Include close in write timing 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Include fsync in write timing 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: File size set to 1048576 kB 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Record Size 16 kB 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-28T02:12:42.772 INFO:tasks.workunit.client.0.smithi106.stdout: Output is in kBytes/sec 2024-03-28T02:12:42.773 INFO:tasks.workunit.client.0.smithi106.stdout: Time Resolution = 0.000001 seconds. 2024-03-28T02:12:42.773 INFO:tasks.workunit.client.0.smithi106.stdout: Processor cache size set to 1024 kBytes. 2024-03-28T02:12:42.773 INFO:tasks.workunit.client.0.smithi106.stdout: Processor cache line size set to 32 bytes. 2024-03-28T02:12:42.773 INFO:tasks.workunit.client.0.smithi106.stdout: File stride size set to 17 * record size. 2024-03-28T02:12:42.773 INFO:tasks.workunit.client.0.smithi106.stdout: Throughput test with 1 process 2024-03-28T02:12:42.773 INFO:tasks.workunit.client.0.smithi106.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-03-28T02:12:43.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:43 smithi106 ceph-mon[31965]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:43 smithi178 ceph-mon[36521]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:43 smithi153 ceph-mon[36228]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:12:44.132 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:12:44.133 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 initial writers = 771438.38 kB/sec 2024-03-28T02:12:44.133 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 initial writers = 771338.41 kB/sec 2024-03-28T02:12:44.133 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 771438.38 kB/sec 2024-03-28T02:12:44.133 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 771438.38 kB/sec 2024-03-28T02:12:44.133 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 771438.38 kB/sec 2024-03-28T02:12:44.133 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 1048576.00 kB 2024-03-28T02:12:45.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:45 smithi106 ceph-mon[31965]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:45 smithi178 ceph-mon[36521]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:45.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:45 smithi153 ceph-mon[36228]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:47.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:47 smithi106 ceph-mon[31965]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:47 smithi178 ceph-mon[36521]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:47.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:47 smithi153 ceph-mon[36228]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T02:12:48.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:48 smithi106 ceph-mon[31965]: mgrmap e25: x(active, since 91s), standbys: z, y 2024-03-28T02:12:48.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:48 smithi178 ceph-mon[36521]: mgrmap e25: x(active, since 91s), standbys: z, y 2024-03-28T02:12:48.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:48 smithi153 ceph-mon[36228]: mgrmap e25: x(active, since 91s), standbys: z, y 2024-03-28T02:12:49.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:49 smithi106 ceph-mon[31965]: pgmap v49: 129 pgs: 129 active+clean; 1.0 GiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 MiB/s wr, 36 op/s 2024-03-28T02:12:49.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:49 smithi178 ceph-mon[36521]: pgmap v49: 129 pgs: 129 active+clean; 1.0 GiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 MiB/s wr, 36 op/s 2024-03-28T02:12:49.970 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:12:49.970 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 rewriters = 832368.25 kB/sec 2024-03-28T02:12:49.970 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 rewriters = 832233.60 kB/sec 2024-03-28T02:12:49.970 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 832368.25 kB/sec 2024-03-28T02:12:49.970 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 832368.25 kB/sec 2024-03-28T02:12:49.970 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 832368.25 kB/sec 2024-03-28T02:12:49.970 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 1048576.00 kB 2024-03-28T02:12:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:49 smithi153 ceph-mon[36228]: pgmap v49: 129 pgs: 129 active+clean; 1.0 GiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 MiB/s wr, 36 op/s 2024-03-28T02:12:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.711+0000 7fcfea0e9640 1 -- 172.21.15.106:0/3644789262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4102010 msgr2=0x7fcfe410e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.711+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/3644789262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4102010 0x7fcfe410e520 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009cd0 tx=0x7fcfdc02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.711+0000 7fcfea0e9640 1 -- 172.21.15.106:0/3644789262 shutdown_connections 2024-03-28T02:12:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.711+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/3644789262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4102010 0x7fcfe410e520 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.711+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/3644789262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfe4101650 0x7fcfe4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.711+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/3644789262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe4107660 0x7fcfe4107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.711+0000 7fcfea0e9640 1 -- 172.21.15.106:0/3644789262 >> 172.21.15.106:0/3644789262 conn(0x7fcfe40fd160 msgr2=0x7fcfe40ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.711+0000 7fcfea0e9640 1 -- 172.21.15.106:0/3644789262 shutdown_connections 2024-03-28T02:12:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.711+0000 7fcfea0e9640 1 -- 172.21.15.106:0/3644789262 wait complete. 2024-03-28T02:12:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfea0e9640 1 Processor -- start 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfea0e9640 1 -- start start 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfea0e9640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfe4101650 0x7fcfe4197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfea0e9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe4102010 0x7fcfe4198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfea0e9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4107660 0x7fcfe419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfea0e9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4111a70 con 0x7fcfe4107660 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfea0e9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41118f0 con 0x7fcfe4102010 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfea0e9640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4111bf0 con 0x7fcfe4101650 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfe3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4107660 0x7fcfe419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfe4101650 0x7fcfe4197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfe3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4107660 0x7fcfe419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51266/0 (socket says 172.21.15.106:51266) 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfe3fff640 1 -- 172.21.15.106:0/2914715391 learned_addr learned my addr 172.21.15.106:0/2914715391 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.712+0000 7fcfe2ffd640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe4102010 0x7fcfe4198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe37fe640 1 -- 172.21.15.106:0/2914715391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe4102010 msgr2=0x7fcfe4198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe4102010 0x7fcfe4198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe37fe640 1 -- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4107660 msgr2=0x7fcfe419c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4107660 0x7fcfe419c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe37fe640 1 -- 172.21.15.106:0/2914715391 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe419ce40 con 0x7fcfe4101650 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe3fff640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4107660 0x7fcfe419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfe4101650 0x7fcfe4197e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9f0 tx=0x7fcfd400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe0ff9640 1 -- 172.21.15.106:0/2914715391 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400ccc0 con 0x7fcfe4101650 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfea0e9640 1 -- 172.21.15.106:0/2914715391 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe419d130 con 0x7fcfe4101650 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe0ff9640 1 -- 172.21.15.106:0/2914715391 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4010910 con 0x7fcfe4101650 2024-03-28T02:12:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfe0ff9640 1 -- 172.21.15.106:0/2914715391 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4018960 con 0x7fcfe4101650 2024-03-28T02:12:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.713+0000 7fcfea0e9640 1 -- 172.21.15.106:0/2914715391 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe41a49c0 con 0x7fcfe4101650 2024-03-28T02:12:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfe0ff9640 1 -- 172.21.15.106:0/2914715391 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fcfd4010ac0 con 0x7fcfe4101650 2024-03-28T02:12:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfecda8740 1 Processor -- start 2024-03-28T02:12:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfecda8740 1 -- start start 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfe0ff9640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fcfcc0a2a10 0x7fcfcc0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 0x55e8d29f58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 0x55e8d29f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfecda8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d2950b10 con 0x55e8d29ec710 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfecda8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d28f9900 con 0x55e8d29f5e00 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfecda8740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d294cbb0 con 0x55e8d29ed030 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfe3fff640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fcfcc0a2a10 0x7fcfcc0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.716+0000 7fcfe2ffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 0x55e8d29f58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe2ffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 0x55e8d29f58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:54792/0 (socket says 172.21.15.106:54792) 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29ecaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51272/0 (socket says 172.21.15.106:51272) 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe2ffd640 1 -- 172.21.15.106:0/1246628981 learned_addr learned my addr 172.21.15.106:0/1246628981 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe3fff640 1 --2- 172.21.15.106:0/1246628981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 0x55e8d29f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe0ff9640 1 -- 172.21.15.106:0/2914715391 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fcfd40cf460 con 0x7fcfe4101650 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe37fe640 1 -- 172.21.15.106:0/1246628981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 msgr2=0x55e8d29f58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/1246628981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 0x55e8d29f58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe37fe640 1 -- 172.21.15.106:0/1246628981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 msgr2=0x55e8d29f81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/1246628981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 0x55e8d29f81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe37fe640 1 -- 172.21.15.106:0/1246628981 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8d2665210 con 0x55e8d29ec710 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe3fff640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fcfcc0a2a10 0x7fcfcc0a4ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009a60 tx=0x7fcfdc032420 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/1246628981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29ecaf0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40cc8d0 tx=0x7fcfd40145e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614beffb7bc2bd3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfc9ffb640 1 -- 172.21.15.106:0/1246628981 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd409b890 con 0x55e8d29ec710 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfc9ffb640 1 -- 172.21.15.106:0/1246628981 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd409f930 con 0x55e8d29ec710 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.717+0000 7fcfc9ffb640 1 -- 172.21.15.106:0/1246628981 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd40ccb50 con 0x55e8d29ec710 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 -- 172.21.15.106:0/1246628981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 msgr2=0x55e8d29ecaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 --2- 172.21.15.106:0/1246628981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29ecaf0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40cc8d0 tx=0x7fcfd40145e0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 -- 172.21.15.106:0/1246628981 shutdown_connections 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 --2- 172.21.15.106:0/1246628981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 0x55e8d29f81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 --2- 172.21.15.106:0/1246628981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 0x55e8d29f58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 --2- 172.21.15.106:0/1246628981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29ecaf0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 -- 172.21.15.106:0/1246628981 >> 172.21.15.106:0/1246628981 conn(0x55e8d29e7030 msgr2=0x55e8d29e7410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 -- 172.21.15.106:0/1246628981 shutdown_connections 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 -- 172.21.15.106:0/1246628981 wait complete. 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 -- start start 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.718+0000 7fcfecda8740 1 Processor -- start 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ec710 0x55e8d29f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ed030 0x55e8d29f3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29f5e00 0x55e8d29f3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfecda8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d29f89d0 con 0x55e8d29ed030 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfecda8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d290cd50 con 0x55e8d29ec710 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfecda8740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d29f8b50 con 0x55e8d29f5e00 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfecda8740 10 client.0 _ll_register_callbacks cb 0x55e8d290c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:12:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ed030 0x55e8d29f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ec710 0x55e8d29f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ed030 0x55e8d29f3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51282/0 (socket says 172.21.15.106:51282) 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe2ffd640 1 -- 172.21.15.106:0/2722357386 learned_addr learned my addr 172.21.15.106:0/2722357386 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe3fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29f5e00 0x55e8d29f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe2ffd640 1 -- 172.21.15.106:0/2722357386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29f5e00 msgr2=0x55e8d29f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe2ffd640 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29f5e00 0x55e8d29f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe2ffd640 1 -- 172.21.15.106:0/2722357386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ec710 msgr2=0x55e8d29f2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe2ffd640 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ec710 0x55e8d29f2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe2ffd640 1 -- 172.21.15.106:0/2722357386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e8d2731620 con 0x55e8d29ed030 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ec710 0x55e8d29f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfe2ffd640 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ed030 0x55e8d29f3450 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8012740 tx=0x7fcfd8012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ef8bf699e04b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/2722357386 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8004050 con 0x55e8d29ed030 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/2722357386 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fcfd80041f0 con 0x55e8d29ed030 2024-03-28T02:12:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.719+0000 7fcfecda8740 1 -- 172.21.15.106:0/2722357386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfd409f930 con 0x55e8d29ed030 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/2722357386 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfd8025050 con 0x55e8d29ed030 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfecda8740 10 client.15747 fetch_fsmap learned FSMap version 29 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfecda8740 1 -- 172.21.15.106:0/2722357386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e8d2a04dc0 con 0x55e8d29ed030 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/2722357386 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd409f930 con 0x55e8d29ed030 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/2722357386 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fcfd8003b10 con 0x55e8d29ed030 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfecda8740 10 client.15747 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfecda8740 10 client.15747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfecda8740 20 client.15747 populate_metadata read hostname 'smithi106' 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfecda8740 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e8d2a06280 0x55e8d2a08670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfecda8740 4 client.15747 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfecda8740 1 -- 172.21.15.106:0/2722357386 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e8d295e050 con 0x55e8d2a06280 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.720+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e8d2a06280 0x55e8d2a08670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfc2ffd640 10 client.15747 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e8d2a06280 0x55e8d2a08670 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2af39ccb13a7e309 server_cookie=844dd4f7127c47a7 in_seq=0 out_seq=0 2024-03-28T02:12:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/2722357386 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e8d295e050 con 0x55e8d2a06280 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfc2ffd640 10 client.15747 handle_command_reply: tid=0 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 client.15747 shutdown 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 -- 172.21.15.106:0/2722357386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ed030 msgr2=0x55e8d29f3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ed030 0x55e8d29f3450 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8012740 tx=0x7fcfd8012c00 comp rx=0 tx=0).stop 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 -- 172.21.15.106:0/2722357386 shutdown_connections 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e8d2a06280 0x55e8d2a08670 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29f5e00 0x55e8d29f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ed030 0x55e8d29f3450 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 --2- 172.21.15.106:0/2722357386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ec710 0x55e8d29f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfc2ffd640 0 client.15747 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 -- 172.21.15.106:0/2722357386 >> 172.21.15.106:0/2722357386 conn(0x55e8d29e7030 msgr2=0x55e8d29e8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 -- 172.21.15.106:0/2722357386 shutdown_connections 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.721+0000 7fcfecda8740 1 -- 172.21.15.106:0/2722357386 wait complete. 2024-03-28T02:12:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.722+0000 7fcfecda8740 20 client.15747 trim_cache size 0 max 16384 2024-03-28T02:12:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfecda8740 1 Processor -- start 2024-03-28T02:12:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfecda8740 1 -- start start 2024-03-28T02:12:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ec710 0x55e8d29f5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ed030 0x55e8d29e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29f5e00 0x55e8d2a01260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfecda8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41ae7e0 con 0x55e8d29f5e00 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfecda8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41b0160 con 0x55e8d29ed030 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfecda8740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41aee40 con 0x55e8d29ec710 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfe3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29f5e00 0x55e8d2a01260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfe3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29f5e00 0x55e8d2a01260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51298/0 (socket says 172.21.15.106:51298) 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfe3fff640 1 -- 172.21.15.106:0/706419983 learned_addr learned my addr 172.21.15.106:0/706419983 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfe2ffd640 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ed030 0x55e8d29e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.735+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ec710 0x55e8d29f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfe3fff640 1 -- 172.21.15.106:0/706419983 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ec710 msgr2=0x55e8d29f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfe3fff640 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ec710 0x55e8d29f5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfe3fff640 1 -- 172.21.15.106:0/706419983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ed030 msgr2=0x55e8d29e8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfe3fff640 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ed030 0x55e8d29e8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfe3fff640 1 -- 172.21.15.106:0/706419983 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8d29f8560 con 0x55e8d29f5e00 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfe2ffd640 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ed030 0x55e8d29e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfe3fff640 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29f5e00 0x55e8d2a01260 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009ca0 tx=0x7fcfdc02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21fdc987b17d3dd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfc97fa640 1 -- 172.21.15.106:0/706419983 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc039040 con 0x55e8d29f5e00 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfc97fa640 1 -- 172.21.15.106:0/706419983 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc003e50 con 0x55e8d29f5e00 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.736+0000 7fcfc97fa640 1 -- 172.21.15.106:0/706419983 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc004050 con 0x55e8d29f5e00 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 -- 172.21.15.106:0/706419983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29f5e00 msgr2=0x55e8d2a01260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29f5e00 0x55e8d2a01260 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009ca0 tx=0x7fcfdc02e050 comp rx=0 tx=0).stop 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 -- 172.21.15.106:0/706419983 shutdown_connections 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29f5e00 0x55e8d2a01260 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29ed030 0x55e8d29e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 --2- 172.21.15.106:0/706419983 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ec710 0x55e8d29f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 -- 172.21.15.106:0/706419983 >> 172.21.15.106:0/706419983 conn(0x55e8d29e7030 msgr2=0x55e8d29e8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 -- 172.21.15.106:0/706419983 shutdown_connections 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 -- 172.21.15.106:0/706419983 wait complete. 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 -- start start 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 Processor -- start 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29efd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.737+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 0x55e8d29f0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfecda8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 0x55e8d2a60ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfecda8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41afb00 con 0x55e8d29ec710 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfecda8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41af4a0 con 0x55e8d29f5e00 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfecda8740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41ac9a0 con 0x55e8d29ed030 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfecda8740 10 client.0 _ll_register_callbacks cb 0x7fcfd8009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29efd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29efd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51304/0 (socket says 172.21.15.106:51304) 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe37fe640 1 -- 172.21.15.106:0/79147033 learned_addr learned my addr 172.21.15.106:0/79147033 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe2ffd640 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 0x55e8d29f0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe3fff640 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 0x55e8d2a60ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe37fe640 1 -- 172.21.15.106:0/79147033 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 msgr2=0x55e8d29f0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 0x55e8d29f0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe37fe640 1 -- 172.21.15.106:0/79147033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 msgr2=0x55e8d2a60ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 0x55e8d2a60ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe37fe640 1 -- 172.21.15.106:0/79147033 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e8d2a61020 con 0x55e8d29ec710 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfe37fe640 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29efd10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcfd409b590 tx=0x7fcfd4010e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcf6a86957bb88f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/79147033 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd409f930 con 0x55e8d29ec710 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/79147033 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fcfd40b4590 con 0x55e8d29ec710 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfecda8740 1 -- 172.21.15.106:0/79147033 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfdc003e50 con 0x55e8d29ec710 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/79147033 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfd4086110 con 0x55e8d29ec710 2024-03-28T02:12:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfecda8740 10 client.15753 fetch_fsmap learned FSMap version 29 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfecda8740 1 -- 172.21.15.106:0/79147033 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e8d2a622c0 con 0x55e8d29ec710 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.738+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/79147033 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc003e50 con 0x55e8d29ec710 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/79147033 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fcfd40b41b0 con 0x55e8d29ec710 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfecda8740 10 client.15753 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfecda8740 10 client.15753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfecda8740 20 client.15753 populate_metadata read hostname 'smithi106' 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfecda8740 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e8d2a625b0 0x55e8d2a649a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfecda8740 4 client.15753 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfecda8740 1 -- 172.21.15.106:0/79147033 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e8d29583a0 con 0x55e8d2a625b0 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfe2ffd640 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e8d2a625b0 0x55e8d2a649a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfe2ffd640 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e8d2a625b0 0x55e8d2a649a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c7ed448034c9078 server_cookie=c41b8ffc4aa8ec7d in_seq=0 out_seq=0 2024-03-28T02:12:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.739+0000 7fcfc2ffd640 10 client.15753 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfc2ffd640 1 -- 172.21.15.106:0/79147033 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e8d29583a0 con 0x55e8d2a625b0 2024-03-28T02:12:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfc2ffd640 10 client.15753 handle_command_reply: tid=0 2024-03-28T02:12:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 client.15753 shutdown 2024-03-28T02:12:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 -- 172.21.15.106:0/79147033 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 msgr2=0x55e8d29efd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29efd10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcfd409b590 tx=0x7fcfd4010e30 comp rx=0 tx=0).stop 2024-03-28T02:12:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 -- 172.21.15.106:0/79147033 shutdown_connections 2024-03-28T02:12:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e8d2a625b0 0x55e8d2a649a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e8d29f5e00 0x55e8d2a60ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e8d29ed030 0x55e8d29f0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfc2ffd640 0 client.15753 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:12:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 --2- 172.21.15.106:0/79147033 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e8d29ec710 0x55e8d29efd10 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 -- 172.21.15.106:0/79147033 >> 172.21.15.106:0/79147033 conn(0x55e8d29e7030 msgr2=0x55e8d29e6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 -- 172.21.15.106:0/79147033 shutdown_connections 2024-03-28T02:12:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 1 -- 172.21.15.106:0/79147033 wait complete. 2024-03-28T02:12:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.748+0000 7fcfecda8740 20 client.15753 trim_cache size 0 max 16384 2024-03-28T02:12:50.743 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:12:50.743 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:12:50.743 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:12:50.743 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 -- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fcfcc0a2a10 msgr2=0x7fcfcc0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fcfcc0a2a10 0x7fcfcc0a4ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009a60 tx=0x7fcfdc032420 comp rx=0 tx=0).stop 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 -- 172.21.15.106:0/2914715391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfe4101650 msgr2=0x7fcfe4197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfe4101650 0x7fcfe4197e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9f0 tx=0x7fcfd400eeb0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 -- 172.21.15.106:0/2914715391 shutdown_connections 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fcfcc0a2a10 0x7fcfcc0a4ed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe4107660 0x7fcfe419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe4102010 0x7fcfe4198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 --2- 172.21.15.106:0/2914715391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfe4101650 0x7fcfe4197e50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 -- 172.21.15.106:0/2914715391 >> 172.21.15.106:0/2914715391 conn(0x7fcfe40fd160 msgr2=0x7fcfe40fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 -- 172.21.15.106:0/2914715391 shutdown_connections 2024-03-28T02:12:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:12:50.749+0000 7fcfea0e9640 1 -- 172.21.15.106:0/2914715391 wait complete. 2024-03-28T02:12:51.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:51 smithi106 ceph-mon[31965]: pgmap v50: 129 pgs: 129 active+clean; 1.0 GiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 85 MiB/s wr, 32 op/s 2024-03-28T02:12:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:51 smithi178 ceph-mon[36521]: pgmap v50: 129 pgs: 129 active+clean; 1.0 GiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 85 MiB/s wr, 32 op/s 2024-03-28T02:12:51.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:51 smithi153 ceph-mon[36228]: pgmap v50: 129 pgs: 129 active+clean; 1.0 GiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 85 MiB/s wr, 32 op/s 2024-03-28T02:12:52.637 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:12:52.637 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 readers = 6816058.00 kB/sec 2024-03-28T02:12:52.637 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 readers = 6806416.71 kB/sec 2024-03-28T02:12:52.637 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 6816058.00 kB/sec 2024-03-28T02:12:52.637 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 6816058.00 kB/sec 2024-03-28T02:12:52.638 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 6816058.00 kB/sec 2024-03-28T02:12:52.638 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 1048576.00 kB 2024-03-28T02:12:53.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:53 smithi106 ceph-mon[31965]: pgmap v51: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 62 op/s 2024-03-28T02:12:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:53 smithi178 ceph-mon[36521]: pgmap v51: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 62 op/s 2024-03-28T02:12:53.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:53 smithi153 ceph-mon[36228]: pgmap v51: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 62 op/s 2024-03-28T02:12:54.903 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:12:54.903 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 re-readers = 6904747.00 kB/sec 2024-03-28T02:12:54.903 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 re-readers = 6895167.09 kB/sec 2024-03-28T02:12:54.903 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 6904747.00 kB/sec 2024-03-28T02:12:54.903 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 6904747.00 kB/sec 2024-03-28T02:12:54.903 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 6904747.00 kB/sec 2024-03-28T02:12:54.903 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 1048576.00 kB 2024-03-28T02:12:55.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:55 smithi153 ceph-mon[36228]: pgmap v52: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-03-28T02:12:55.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:55 smithi106 ceph-mon[31965]: pgmap v52: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-03-28T02:12:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:55 smithi178 ceph-mon[36521]: pgmap v52: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-03-28T02:12:57.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:57 smithi106 ceph-mon[31965]: pgmap v53: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-03-28T02:12:57.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:57 smithi178 ceph-mon[36521]: pgmap v53: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-03-28T02:12:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:57 smithi153 ceph-mon[36228]: pgmap v53: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-03-28T02:12:59.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:12:59 smithi106 ceph-mon[31965]: pgmap v54: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 61 op/s 2024-03-28T02:12:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:12:59 smithi178 ceph-mon[36521]: pgmap v54: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 61 op/s 2024-03-28T02:12:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:12:59 smithi153 ceph-mon[36228]: pgmap v54: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 61 op/s 2024-03-28T02:13:01.243 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:01.243 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:01.243 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:01.243 INFO:tasks.workunit.client.0.smithi106.stdout:iozone test complete. 2024-03-28T02:13:01.243 INFO:tasks.workunit.client.0.smithi106.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Iozone: Performance Test of File I/O 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Version $Revision: 3.494 $ 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Compiled for 64 bit mode. 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Build: linux-AMD64 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-28T02:13:01.309 INFO:tasks.workunit.client.0.smithi106.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Run began: Thu Mar 28 02:13:01 2024 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Include close in write timing 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Include fsync in write timing 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: File size set to 1048576 kB 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Record Size 1024 kB 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Output is in kBytes/sec 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Time Resolution = 0.000001 seconds. 2024-03-28T02:13:01.310 INFO:tasks.workunit.client.0.smithi106.stdout: Processor cache size set to 1024 kBytes. 2024-03-28T02:13:01.311 INFO:tasks.workunit.client.0.smithi106.stdout: Processor cache line size set to 32 bytes. 2024-03-28T02:13:01.311 INFO:tasks.workunit.client.0.smithi106.stdout: File stride size set to 17 * record size. 2024-03-28T02:13:01.311 INFO:tasks.workunit.client.0.smithi106.stdout: Throughput test with 1 process 2024-03-28T02:13:01.311 INFO:tasks.workunit.client.0.smithi106.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-03-28T02:13:01.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:01 smithi106 ceph-mon[31965]: pgmap v55: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 85 MiB/s wr, 33 op/s 2024-03-28T02:13:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:01 smithi178 ceph-mon[36521]: pgmap v55: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 85 MiB/s wr, 33 op/s 2024-03-28T02:13:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:01 smithi153 ceph-mon[36228]: pgmap v55: 129 pgs: 129 active+clean; 1.0 GiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 85 MiB/s wr, 33 op/s 2024-03-28T02:13:02.652 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:02.652 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 initial writers = 799335.25 kB/sec 2024-03-28T02:13:02.652 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 initial writers = 799195.70 kB/sec 2024-03-28T02:13:02.652 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 799335.25 kB/sec 2024-03-28T02:13:02.652 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 799335.25 kB/sec 2024-03-28T02:13:02.652 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 799335.25 kB/sec 2024-03-28T02:13:02.652 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 1048576.00 kB 2024-03-28T02:13:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.835+0000 7f257e4c6640 1 -- 172.21.15.106:0/531005353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2578106610 msgr2=0x7f2578069f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.835+0000 7f257e4c6640 1 --2- 172.21.15.106:0/531005353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2578106610 0x7f2578069f40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2568009930 tx=0x7f256802bc30 comp rx=0 tx=0).stop 2024-03-28T02:13:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.835+0000 7f257e4c6640 1 -- 172.21.15.106:0/531005353 shutdown_connections 2024-03-28T02:13:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.835+0000 7f257e4c6640 1 --2- 172.21.15.106:0/531005353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2578104180 0x7f2578104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.835+0000 7f257e4c6640 1 --2- 172.21.15.106:0/531005353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f257806a480 0x7f257806a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.835+0000 7f257e4c6640 1 --2- 172.21.15.106:0/531005353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2578106610 0x7f2578069f40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.835+0000 7f257e4c6640 1 -- 172.21.15.106:0/531005353 >> 172.21.15.106:0/531005353 conn(0x7f2578077660 msgr2=0x7f2578077a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.835+0000 7f257e4c6640 1 -- 172.21.15.106:0/531005353 shutdown_connections 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.835+0000 7f257e4c6640 1 -- 172.21.15.106:0/531005353 wait complete. 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f257e4c6640 1 Processor -- start 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f257e4c6640 1 -- start start 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f257e4c6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f257806a480 0x7f257819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f257e4c6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2578104180 0x7f257819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f257e4c6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2578106610 0x7f25781a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f257e4c6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2578118030 con 0x7f257806a480 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f257e4c6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2578117eb0 con 0x7f2578104180 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f257e4c6640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781181b0 con 0x7f2578106610 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f2577fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f257806a480 0x7f257819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f2577fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f257806a480 0x7f257819e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41568/0 (socket says 172.21.15.106:41568) 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f2577fff640 1 -- 172.21.15.106:0/611601492 learned_addr learned my addr 172.21.15.106:0/611601492 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f25777fe640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2578104180 0x7f257819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.836+0000 7f257ca63640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2578106610 0x7f25781a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f2577fff640 1 -- 172.21.15.106:0/611601492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2578106610 msgr2=0x7f25781a2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f2577fff640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2578106610 0x7f25781a2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f2577fff640 1 -- 172.21.15.106:0/611601492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2578104180 msgr2=0x7f257819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f2577fff640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2578104180 0x7f257819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f2577fff640 1 -- 172.21.15.106:0/611601492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25781a3420 con 0x7f257806a480 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f2577fff640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f257806a480 0x7f257819e430 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2568009230 tx=0x7f256802cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f25757fa640 1 -- 172.21.15.106:0/611601492 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568041070 con 0x7f257806a480 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f25757fa640 1 -- 172.21.15.106:0/611601492 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2568004450 con 0x7f257806a480 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f257e4c6640 1 -- 172.21.15.106:0/611601492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25781a36b0 con 0x7f257806a480 2024-03-28T02:13:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f25757fa640 1 -- 172.21.15.106:0/611601492 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568034440 con 0x7f257806a480 2024-03-28T02:13:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.837+0000 7f257e4c6640 1 -- 172.21.15.106:0/611601492 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25781aafa0 con 0x7f257806a480 2024-03-28T02:13:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.839+0000 7f25757fa640 1 -- 172.21.15.106:0/611601492 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f2568044070 con 0x7f257806a480 2024-03-28T02:13:03.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:03 smithi106 ceph-mon[31965]: pgmap v56: 129 pgs: 129 active+clean; 1.8 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 150 MiB/s wr, 70 op/s 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f2581190740 1 Processor -- start 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f2581190740 1 -- start start 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25757fa640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f25600a2a10 0x7f25600a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f2581190740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8732af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f257ca63640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f25600a2a10 0x7f25600a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f2581190740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8733030 0x5634b873b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f2581190740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b873be00 0x5634b873e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f2581190740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634b8696b10 con 0x5634b873be00 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f2581190740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634b863f900 con 0x5634b8733030 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25757fa640 1 -- 172.21.15.106:0/611601492 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f25680f0af0 con 0x7f257806a480 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f2581190740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634b8692bb0 con 0x5634b8732710 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25777fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8733030 0x5634b873b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25777fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8733030 0x5634b873b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:44410/0 (socket says 172.21.15.106:44410) 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25777fe640 1 -- 172.21.15.106:0/243119542 learned_addr learned my addr 172.21.15.106:0/243119542 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f257ca63640 1 --2- 172.21.15.106:0/243119542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b873be00 0x5634b873e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f2577fff640 1 --2- 172.21.15.106:0/243119542 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8732af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25777fe640 1 -- 172.21.15.106:0/243119542 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 msgr2=0x5634b8732af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25777fe640 1 --2- 172.21.15.106:0/243119542 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8732af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25777fe640 1 -- 172.21.15.106:0/243119542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b873be00 msgr2=0x5634b873e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25777fe640 1 --2- 172.21.15.106:0/243119542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b873be00 0x5634b873e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f25777fe640 1 -- 172.21.15.106:0/243119542 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634b83ab210 con 0x5634b8733030 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.840+0000 7f257ca63640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f25600a2a10 0x7f25600a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2570005fd0 tx=0x7f2570013ae0 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.841+0000 7f25777fe640 1 --2- 172.21.15.106:0/243119542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8733030 0x5634b873b8c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f256c00bd10 tx=0x7f256c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cf4fdb23ca40c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.841+0000 7f255e7fc640 1 -- 172.21.15.106:0/243119542 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c005020 con 0x5634b8733030 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.841+0000 7f255e7fc640 1 -- 172.21.15.106:0/243119542 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f256c0047a0 con 0x5634b8733030 2024-03-28T02:13:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.841+0000 7f255e7fc640 1 -- 172.21.15.106:0/243119542 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c011730 con 0x5634b8733030 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.841+0000 7f2581190740 1 -- 172.21.15.106:0/243119542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8733030 msgr2=0x5634b873b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.841+0000 7f2581190740 1 --2- 172.21.15.106:0/243119542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8733030 0x5634b873b8c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f256c00bd10 tx=0x7f256c0107a0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 -- 172.21.15.106:0/243119542 shutdown_connections 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 --2- 172.21.15.106:0/243119542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b873be00 0x5634b873e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 --2- 172.21.15.106:0/243119542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8733030 0x5634b873b8c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 --2- 172.21.15.106:0/243119542 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8732af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 -- 172.21.15.106:0/243119542 >> 172.21.15.106:0/243119542 conn(0x5634b872d030 msgr2=0x5634b872d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 -- 172.21.15.106:0/243119542 shutdown_connections 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 -- 172.21.15.106:0/243119542 wait complete. 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 -- start start 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 Processor -- start 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8738e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b87393d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 0x5634b8739910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634b873ea30 con 0x5634b8733030 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634b8652d50 con 0x5634b873be00 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634b873ebb0 con 0x5634b8732710 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f2581190740 10 client.0 _ll_register_callbacks cb 0x5634b8652720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.842+0000 7f25777fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b87393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2577fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8738e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f257ca63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 0x5634b8739910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2577fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8738e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:53542/0 (socket says 172.21.15.106:53542) 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2577fff640 1 -- 172.21.15.106:0/2285719360 learned_addr learned my addr 172.21.15.106:0/2285719360 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f25777fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b87393d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41588/0 (socket says 172.21.15.106:41588) 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2577fff640 1 -- 172.21.15.106:0/2285719360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 msgr2=0x5634b8739910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2577fff640 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 0x5634b8739910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2577fff640 1 -- 172.21.15.106:0/2285719360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 msgr2=0x5634b87393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2577fff640 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b87393d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2577fff640 1 -- 172.21.15.106:0/2285719360 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634b8477620 con 0x5634b8732710 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f257ca63640 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 0x5634b8739910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f25777fe640 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b87393d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2577fff640 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8738e90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f25680bcec0 tx=0x7f25680b9300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3298482e0cf51e7e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f25577fe640 10 client.0 ms_handle_connect on v2:172.21.15.178:3300/0 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f25577fe640 1 -- 172.21.15.106:0/2285719360 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256804b040 con 0x5634b8732710 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2581190740 1 -- 172.21.15.106:0/2285719360 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f256c0047a0 con 0x5634b8732710 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f25577fe640 1 -- 172.21.15.106:0/2285719360 <== mon.1 v2:172.21.15.178:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f25680c5190 con 0x5634b8732710 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f25577fe640 1 -- 172.21.15.106:0/2285719360 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2568004c80 con 0x5634b8732710 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2581190740 10 client.25540 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.843+0000 7f2581190740 1 -- 172.21.15.106:0/2285719360 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5634b873eae0 con 0x5634b8732710 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f25577fe640 1 -- 172.21.15.106:0/2285719360 <== mon.1 v2:172.21.15.178:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f256c0047a0 con 0x5634b8732710 2024-03-28T02:13:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f25577fe640 1 -- 172.21.15.106:0/2285719360 <== mon.1 v2:172.21.15.178:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f25680bc380 con 0x5634b8732710 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f2581190740 10 client.25540 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f2581190740 10 client.25540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f2581190740 20 client.25540 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f2581190740 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5634b874c1c0 0x5634b874e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f2581190740 4 client.25540 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f2581190740 1 -- 172.21.15.106:0/2285719360 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5634b86a4050 con 0x5634b874c1c0 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f25777fe640 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5634b874c1c0 0x5634b874e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f25777fe640 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5634b874c1c0 0x5634b874e5b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9c65146af1459 server_cookie=e502d10d23cdd1cd in_seq=0 out_seq=0 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.844+0000 7f25577fe640 10 client.25540 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f25577fe640 1 -- 172.21.15.106:0/2285719360 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5634b86a4050 con 0x5634b874c1c0 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f25577fe640 10 client.25540 handle_command_reply: tid=0 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 client.25540 shutdown 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 -- 172.21.15.106:0/2285719360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 msgr2=0x5634b8738e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8738e90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f25680bcec0 tx=0x7f25680b9300 comp rx=0 tx=0).stop 2024-03-28T02:13:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 -- 172.21.15.106:0/2285719360 shutdown_connections 2024-03-28T02:13:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5634b874c1c0 0x5634b874e5b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 0x5634b8739910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f25577fe640 0 client.25540 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b87393d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 --2- 172.21.15.106:0/2285719360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8732710 0x5634b8738e90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 -- 172.21.15.106:0/2285719360 >> 172.21.15.106:0/2285719360 conn(0x5634b872d030 msgr2=0x5634b872ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 -- 172.21.15.106:0/2285719360 shutdown_connections 2024-03-28T02:13:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 1 -- 172.21.15.106:0/2285719360 wait complete. 2024-03-28T02:13:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.845+0000 7f2581190740 20 client.25540 trim_cache size 0 max 16384 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f2581190740 1 Processor -- start 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f2581190740 1 -- start start 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f2581190740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8732710 0x5634b873b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f2581190740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b872eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f2581190740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b873be00 0x5634b87472c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f2581190740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781b4db0 con 0x5634b8733030 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f2581190740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781b6730 con 0x5634b8732710 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f2581190740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781b5410 con 0x5634b873be00 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f25777fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b872eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f25777fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b872eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41594/0 (socket says 172.21.15.106:41594) 2024-03-28T02:13:03.856 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f2577fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8732710 0x5634b873b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f25777fe640 1 -- 172.21.15.106:0/270895004 learned_addr learned my addr 172.21.15.106:0/270895004 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f257ca63640 1 --2- 172.21.15.106:0/270895004 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b873be00 0x5634b87472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f25777fe640 1 -- 172.21.15.106:0/270895004 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b873be00 msgr2=0x5634b87472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f25777fe640 1 --2- 172.21.15.106:0/270895004 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b873be00 0x5634b87472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f25777fe640 1 -- 172.21.15.106:0/270895004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8732710 msgr2=0x5634b873b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f25777fe640 1 --2- 172.21.15.106:0/270895004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8732710 0x5634b873b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f25777fe640 1 -- 172.21.15.106:0/270895004 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634b8732e60 con 0x5634b8733030 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.859+0000 7f25777fe640 1 --2- 172.21.15.106:0/270895004 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b872eb90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x5634b87321c0 tx=0x7f256c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d46160653c59cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f255dffb640 1 -- 172.21.15.106:0/270895004 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c00be00 con 0x5634b8733030 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f255dffb640 1 -- 172.21.15.106:0/270895004 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f256c0290d0 con 0x5634b8733030 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f255dffb640 1 -- 172.21.15.106:0/270895004 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c029cf0 con 0x5634b8733030 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f2581190740 1 -- 172.21.15.106:0/270895004 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 msgr2=0x5634b872eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f2581190740 1 --2- 172.21.15.106:0/270895004 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b872eb90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x5634b87321c0 tx=0x7f256c00b410 comp rx=0 tx=0).stop 2024-03-28T02:13:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f2581190740 1 -- 172.21.15.106:0/270895004 shutdown_connections 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f2581190740 1 --2- 172.21.15.106:0/270895004 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b873be00 0x5634b87472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f2581190740 1 --2- 172.21.15.106:0/270895004 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8733030 0x5634b872eb90 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f2581190740 1 --2- 172.21.15.106:0/270895004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b8732710 0x5634b873b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f2581190740 1 -- 172.21.15.106:0/270895004 >> 172.21.15.106:0/270895004 conn(0x5634b872d030 msgr2=0x5634b872e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f2581190740 1 -- 172.21.15.106:0/270895004 shutdown_connections 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.860+0000 7f2581190740 1 -- 172.21.15.106:0/270895004 wait complete. 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2581190740 1 -- start start 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2581190740 1 Processor -- start 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2581190740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8732710 0x5634b8735850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2581190740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8733030 0x5634b8735d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2581190740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 0x5634b87a6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2581190740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781b60d0 con 0x5634b8732710 2024-03-28T02:13:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2581190740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781b5a70 con 0x5634b873be00 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2581190740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781b2f70 con 0x5634b8733030 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2581190740 10 client.0 _ll_register_callbacks cb 0x7f2568009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2577fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8732710 0x5634b8735850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2577fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8732710 0x5634b8735850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41598/0 (socket says 172.21.15.106:41598) 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2577fff640 1 -- 172.21.15.106:0/4199566682 learned_addr learned my addr 172.21.15.106:0/4199566682 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f257ca63640 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 0x5634b87a6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f25777fe640 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8733030 0x5634b8735d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2577fff640 1 -- 172.21.15.106:0/4199566682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8733030 msgr2=0x5634b8735d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2577fff640 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8733030 0x5634b8735d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2577fff640 1 -- 172.21.15.106:0/4199566682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 msgr2=0x5634b87a6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2577fff640 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 0x5634b87a6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2577fff640 1 -- 172.21.15.106:0/4199566682 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634b87a6f90 con 0x5634b8732710 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f2577fff640 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8732710 0x5634b8735850 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f25680096c0 tx=0x7f25680b91d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb7fe01ea2d4052 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.861+0000 7f25577fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f25577fe640 1 -- 172.21.15.106:0/4199566682 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568004c80 con 0x5634b8732710 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f25577fe640 1 -- 172.21.15.106:0/4199566682 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f25680bbfb0 con 0x5634b8732710 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f2581190740 1 -- 172.21.15.106:0/4199566682 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f256c0290d0 con 0x5634b8732710 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f25577fe640 1 -- 172.21.15.106:0/4199566682 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25680bcf30 con 0x5634b8732710 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f2581190740 10 client.15771 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f2581190740 1 -- 172.21.15.106:0/4199566682 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5634b87a6f90 con 0x5634b8732710 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f25577fe640 1 -- 172.21.15.106:0/4199566682 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f256c0290d0 con 0x5634b8732710 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f25577fe640 1 -- 172.21.15.106:0/4199566682 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f25680a2070 con 0x5634b8732710 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f2581190740 10 client.15771 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f2581190740 10 client.15771 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f2581190740 20 client.15771 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f2581190740 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5634b87a8470 0x5634b87aa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f2581190740 4 client.15771 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:13:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f2581190740 1 -- 172.21.15.106:0/4199566682 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5634b869e3a0 con 0x5634b87a8470 2024-03-28T02:13:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.862+0000 7f25777fe640 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5634b87a8470 0x5634b87aa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.863+0000 7f25577fe640 10 client.15771 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.863+0000 7f25777fe640 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5634b87a8470 0x5634b87aa860 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5fcf4156c094cc3 server_cookie=151fe2c3ec422ffa in_seq=0 out_seq=0 2024-03-28T02:13:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f25577fe640 1 -- 172.21.15.106:0/4199566682 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5634b869e3a0 con 0x5634b87a8470 2024-03-28T02:13:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f25577fe640 10 client.15771 handle_command_reply: tid=0 2024-03-28T02:13:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 client.15771 shutdown 2024-03-28T02:13:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 -- 172.21.15.106:0/4199566682 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8732710 msgr2=0x5634b8735850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8732710 0x5634b8735850 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f25680096c0 tx=0x7f25680b91d0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 -- 172.21.15.106:0/4199566682 shutdown_connections 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5634b87a8470 0x5634b87aa860 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5634b873be00 0x5634b87a6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5634b8733030 0x5634b8735d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f25577fe640 0 client.15771 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 --2- 172.21.15.106:0/4199566682 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5634b8732710 0x5634b8735850 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 -- 172.21.15.106:0/4199566682 >> 172.21.15.106:0/4199566682 conn(0x5634b872d030 msgr2=0x5634b872ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 -- 172.21.15.106:0/4199566682 shutdown_connections 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 1 -- 172.21.15.106:0/4199566682 wait complete. 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.874+0000 7f2581190740 20 client.15771 trim_cache size 0 max 16384 2024-03-28T02:13:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:13:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:13:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:13:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:13:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 -- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f25600a2a10 msgr2=0x7f25600a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f25600a2a10 0x7f25600a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2570005fd0 tx=0x7f2570013ae0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 -- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f257806a480 msgr2=0x7f257819e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f257806a480 0x7f257819e430 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2568009230 tx=0x7f256802cba0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 -- 172.21.15.106:0/611601492 shutdown_connections 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f25600a2a10 0x7f25600a4ed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2578106610 0x7f25781a2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2578104180 0x7f257819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 --2- 172.21.15.106:0/611601492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f257806a480 0x7f257819e430 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 -- 172.21.15.106:0/611601492 >> 172.21.15.106:0/611601492 conn(0x7f2578077660 msgr2=0x7f25780ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 -- 172.21.15.106:0/611601492 shutdown_connections 2024-03-28T02:13:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:03.875+0000 7f257e4c6640 1 -- 172.21.15.106:0/611601492 wait complete. 2024-03-28T02:13:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:03 smithi178 ceph-mon[36521]: pgmap v56: 129 pgs: 129 active+clean; 1.8 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 150 MiB/s wr, 70 op/s 2024-03-28T02:13:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:03 smithi153 ceph-mon[36228]: pgmap v56: 129 pgs: 129 active+clean; 1.8 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 150 MiB/s wr, 70 op/s 2024-03-28T02:13:05.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:05 smithi106 ceph-mon[31965]: pgmap v57: 129 pgs: 129 active+clean; 1.8 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 65 MiB/s wr, 41 op/s 2024-03-28T02:13:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:05 smithi178 ceph-mon[36521]: pgmap v57: 129 pgs: 129 active+clean; 1.8 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 65 MiB/s wr, 41 op/s 2024-03-28T02:13:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:05 smithi153 ceph-mon[36228]: pgmap v57: 129 pgs: 129 active+clean; 1.8 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 65 MiB/s wr, 41 op/s 2024-03-28T02:13:06.265 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:06.265 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 rewriters = 818373.38 kB/sec 2024-03-28T02:13:06.265 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 rewriters = 818217.49 kB/sec 2024-03-28T02:13:06.265 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 818373.38 kB/sec 2024-03-28T02:13:06.265 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 818373.38 kB/sec 2024-03-28T02:13:06.265 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 818373.38 kB/sec 2024-03-28T02:13:06.266 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 1048576.00 kB 2024-03-28T02:13:07.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:07 smithi106 ceph-mon[31965]: pgmap v58: 129 pgs: 129 active+clean; 1.8 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 65 MiB/s wr, 41 op/s 2024-03-28T02:13:07.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:07 smithi178 ceph-mon[36521]: pgmap v58: 129 pgs: 129 active+clean; 1.8 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 65 MiB/s wr, 41 op/s 2024-03-28T02:13:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:07 smithi153 ceph-mon[36228]: pgmap v58: 129 pgs: 129 active+clean; 1.8 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 65 MiB/s wr, 41 op/s 2024-03-28T02:13:09.442 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:09.442 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 readers = 7485542.00 kB/sec 2024-03-28T02:13:09.442 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 readers = 7472480.51 kB/sec 2024-03-28T02:13:09.442 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 7485542.00 kB/sec 2024-03-28T02:13:09.442 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 7485542.00 kB/sec 2024-03-28T02:13:09.442 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 7485542.00 kB/sec 2024-03-28T02:13:09.442 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 1048576.00 kB 2024-03-28T02:13:09.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:09 smithi106 ceph-mon[31965]: pgmap v59: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 171 MiB/s wr, 98 op/s 2024-03-28T02:13:09.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:09 smithi178 ceph-mon[36521]: pgmap v59: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 171 MiB/s wr, 98 op/s 2024-03-28T02:13:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:09 smithi153 ceph-mon[36228]: pgmap v59: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 171 MiB/s wr, 98 op/s 2024-03-28T02:13:11.646 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:11.647 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 re-readers = 7247697.50 kB/sec 2024-03-28T02:13:11.647 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 re-readers = 7237452.53 kB/sec 2024-03-28T02:13:11.647 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 7247697.50 kB/sec 2024-03-28T02:13:11.647 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 7247697.50 kB/sec 2024-03-28T02:13:11.647 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 7247697.50 kB/sec 2024-03-28T02:13:11.647 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 1048576.00 kB 2024-03-28T02:13:11.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:11 smithi106 ceph-mon[31965]: pgmap v60: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 171 MiB/s wr, 94 op/s 2024-03-28T02:13:11.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:11 smithi178 ceph-mon[36521]: pgmap v60: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 171 MiB/s wr, 94 op/s 2024-03-28T02:13:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:11 smithi153 ceph-mon[36228]: pgmap v60: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 171 MiB/s wr, 94 op/s 2024-03-28T02:13:13.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:13 smithi106 ceph-mon[31965]: pgmap v61: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 171 MiB/s wr, 98 op/s 2024-03-28T02:13:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:13 smithi178 ceph-mon[36521]: pgmap v61: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 171 MiB/s wr, 98 op/s 2024-03-28T02:13:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:13 smithi153 ceph-mon[36228]: pgmap v61: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 171 MiB/s wr, 98 op/s 2024-03-28T02:13:15.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:15 smithi106 ceph-mon[31965]: pgmap v62: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 106 MiB/s wr, 61 op/s 2024-03-28T02:13:15.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:15 smithi178 ceph-mon[36521]: pgmap v62: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 106 MiB/s wr, 61 op/s 2024-03-28T02:13:15.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:15 smithi153 ceph-mon[36228]: pgmap v62: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 106 MiB/s wr, 61 op/s 2024-03-28T02:13:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.969+0000 7fde81313640 1 -- 172.21.15.106:0/492318467 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c1007f0 msgr2=0x7fde7c100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.969+0000 7fde81313640 1 --2- 172.21.15.106:0/492318467 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c1007f0 0x7fde7c100c50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fde64009900 tx=0x7fde6402bc30 comp rx=0 tx=0).stop 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.969+0000 7fde81313640 1 -- 172.21.15.106:0/492318467 shutdown_connections 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.969+0000 7fde81313640 1 --2- 172.21.15.106:0/492318467 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde7c101190 0x7fde7c10d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.969+0000 7fde81313640 1 --2- 172.21.15.106:0/492318467 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c1007f0 0x7fde7c100c50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.969+0000 7fde81313640 1 --2- 172.21.15.106:0/492318467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde7c105520 0x7fde7c105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.969+0000 7fde81313640 1 -- 172.21.15.106:0/492318467 >> 172.21.15.106:0/492318467 conn(0x7fde7c077370 msgr2=0x7fde7c077780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.969+0000 7fde81313640 1 -- 172.21.15.106:0/492318467 shutdown_connections 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.969+0000 7fde81313640 1 -- 172.21.15.106:0/492318467 wait complete. 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde81313640 1 Processor -- start 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde81313640 1 -- start start 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde81313640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde7c1007f0 0x7fde7c196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde81313640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde7c101190 0x7fde7c197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde81313640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c105520 0x7fde7c19b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde81313640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c110a30 con 0x7fde7c105520 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde81313640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1108b0 con 0x7fde7c1007f0 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde81313640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c110bb0 con 0x7fde7c101190 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde7b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c105520 0x7fde7c19b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde7affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde7c1007f0 0x7fde7c196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde7b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c105520 0x7fde7c19b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54192/0 (socket says 172.21.15.106:54192) 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde7affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde7c1007f0 0x7fde7c196e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:42426/0 (socket says 172.21.15.106:42426) 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde73fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde7c101190 0x7fde7c197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.970+0000 7fde7b7fe640 1 -- 172.21.15.106:0/846067279 learned_addr learned my addr 172.21.15.106:0/846067279 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde7b7fe640 1 -- 172.21.15.106:0/846067279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde7c101190 msgr2=0x7fde7c197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde7b7fe640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde7c101190 0x7fde7c197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde7b7fe640 1 -- 172.21.15.106:0/846067279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde7c1007f0 msgr2=0x7fde7c196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde7b7fe640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde7c1007f0 0x7fde7c196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde7b7fe640 1 -- 172.21.15.106:0/846067279 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c19bdd0 con 0x7fde7c105520 2024-03-28T02:13:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde7b7fe640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c105520 0x7fde7c19b6a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fde7400eca0 tx=0x7fde7400b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde78ff9640 1 -- 172.21.15.106:0/846067279 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74004b20 con 0x7fde7c105520 2024-03-28T02:13:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde78ff9640 1 -- 172.21.15.106:0/846067279 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde74004cc0 con 0x7fde7c105520 2024-03-28T02:13:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde78ff9640 1 -- 172.21.15.106:0/846067279 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74017650 con 0x7fde7c105520 2024-03-28T02:13:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde81313640 1 -- 172.21.15.106:0/846067279 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c19c0c0 con 0x7fde7c105520 2024-03-28T02:13:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.971+0000 7fde81313640 1 -- 172.21.15.106:0/846067279 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c069ea0 con 0x7fde7c105520 2024-03-28T02:13:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.973+0000 7fde78ff9640 1 -- 172.21.15.106:0/846067279 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fde74005020 con 0x7fde7c105520 2024-03-28T02:13:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde83fd0740 1 Processor -- start 2024-03-28T02:13:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde83fd0740 1 -- start start 2024-03-28T02:13:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde78ff9640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fde680a2a90 0x7fde680a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771ddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771de030 0x55e2771e68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 0x55e2771e91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde83fd0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e277141b10 con 0x55e2771de030 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde83fd0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2770ea900 con 0x55e2771e6e00 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde83fd0740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27713dbb0 con 0x55e2771dd710 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde73fff640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fde680a2a90 0x7fde680a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde7affd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771ddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde73fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771de030 0x55e2771e68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde7affd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771ddaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:38698/0 (socket says 172.21.15.106:38698) 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde7affd640 1 -- 172.21.15.106:0/675361377 learned_addr learned my addr 172.21.15.106:0/675361377 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.974+0000 7fde7b7fe640 1 --2- 172.21.15.106:0/675361377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 0x55e2771e91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde78ff9640 1 -- 172.21.15.106:0/846067279 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fde740d0ba0 con 0x7fde7c105520 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde7affd640 1 -- 172.21.15.106:0/675361377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 msgr2=0x55e2771e91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde7affd640 1 --2- 172.21.15.106:0/675361377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 0x55e2771e91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde7affd640 1 -- 172.21.15.106:0/675361377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771de030 msgr2=0x55e2771e68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde7affd640 1 --2- 172.21.15.106:0/675361377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771de030 0x55e2771e68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde7affd640 1 -- 172.21.15.106:0/675361377 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e276e56210 con 0x55e2771dd710 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde73fff640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fde680a2a90 0x7fde680a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fde64006010 tx=0x7fde64005e40 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde7affd640 1 --2- 172.21.15.106:0/675361377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771ddaf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009cf0 tx=0x7fde6c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b0b4a1b3830f2d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde717fa640 1 -- 172.21.15.106:0/675361377 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c004230 con 0x55e2771dd710 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde717fa640 1 -- 172.21.15.106:0/675361377 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde6c0043d0 con 0x55e2771dd710 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.975+0000 7fde717fa640 1 -- 172.21.15.106:0/675361377 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0198c0 con 0x55e2771dd710 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 -- 172.21.15.106:0/675361377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 msgr2=0x55e2771ddaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 --2- 172.21.15.106:0/675361377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771ddaf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009cf0 tx=0x7fde6c011810 comp rx=0 tx=0).stop 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 -- 172.21.15.106:0/675361377 shutdown_connections 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 --2- 172.21.15.106:0/675361377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 0x55e2771e91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 --2- 172.21.15.106:0/675361377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771de030 0x55e2771e68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 --2- 172.21.15.106:0/675361377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771ddaf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 -- 172.21.15.106:0/675361377 >> 172.21.15.106:0/675361377 conn(0x55e2771d8030 msgr2=0x55e2771d8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 -- 172.21.15.106:0/675361377 shutdown_connections 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 -- 172.21.15.106:0/675361377 wait complete. 2024-03-28T02:13:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 -- start start 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.976+0000 7fde83fd0740 1 Processor -- start 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771e3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771e4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771e4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde83fd0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2771e9a30 con 0x55e2771e6e00 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde83fd0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2770fdd50 con 0x55e2771de030 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde83fd0740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2771e9bb0 con 0x55e2771dd710 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde83fd0740 10 client.0 _ll_register_callbacks cb 0x55e2770fd720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde7b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771e4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde73fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771e4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde7affd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771e3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde73fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771e4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:42442/0 (socket says 172.21.15.106:42442) 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde7b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771e4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54200/0 (socket says 172.21.15.106:54200) 2024-03-28T02:13:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde73fff640 1 -- 172.21.15.106:0/2765297076 learned_addr learned my addr 172.21.15.106:0/2765297076 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde73fff640 1 -- 172.21.15.106:0/2765297076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 msgr2=0x55e2771e3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde73fff640 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771e3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde73fff640 1 -- 172.21.15.106:0/2765297076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 msgr2=0x55e2771e4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde73fff640 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771e4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde73fff640 1 -- 172.21.15.106:0/2765297076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e276f22620 con 0x55e2771de030 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde7b7fe640 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771e4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde7affd640 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771e3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde73fff640 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771e4340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde64035000 tx=0x7fde6402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f38982f6c3c80c58 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.977+0000 7fde627fc640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde627fc640 1 -- 172.21.15.106:0/2765297076 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64004ad0 con 0x55e2771de030 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde83fd0740 1 -- 172.21.15.106:0/2765297076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde6c0043d0 con 0x55e2771de030 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde627fc640 1 -- 172.21.15.106:0/2765297076 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fde64004ec0 con 0x55e2771de030 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde627fc640 1 -- 172.21.15.106:0/2765297076 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde64004c70 con 0x55e2771de030 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde83fd0740 10 client.25637 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde83fd0740 1 -- 172.21.15.106:0/2765297076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e2771f5ca0 con 0x55e2771de030 2024-03-28T02:13:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde627fc640 1 -- 172.21.15.106:0/2765297076 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde6c0043d0 con 0x55e2771de030 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde627fc640 1 -- 172.21.15.106:0/2765297076 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fde64003cb0 con 0x55e2771de030 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde83fd0740 10 client.25637 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde83fd0740 10 client.25637 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde83fd0740 20 client.25637 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde83fd0740 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e2771f7110 0x55e2771f9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde83fd0740 4 client.25637 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde83fd0740 1 -- 172.21.15.106:0/2765297076 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e27714f050 con 0x55e2771f7110 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.978+0000 7fde7affd640 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e2771f7110 0x55e2771f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.979+0000 7fde627fc640 10 client.25637 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.979+0000 7fde7affd640 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e2771f7110 0x55e2771f9500 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc1039ba4a3e3f92 server_cookie=6a29443c4dda227b in_seq=0 out_seq=0 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.979+0000 7fde627fc640 1 -- 172.21.15.106:0/2765297076 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e27714f050 con 0x55e2771f7110 2024-03-28T02:13:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.979+0000 7fde627fc640 10 client.25637 handle_command_reply: tid=0 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.979+0000 7fde83fd0740 1 client.25637 shutdown 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 -- 172.21.15.106:0/2765297076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 msgr2=0x55e2771e4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771e4340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde64035000 tx=0x7fde6402bc30 comp rx=0 tx=0).stop 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 -- 172.21.15.106:0/2765297076 shutdown_connections 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e2771f7110 0x55e2771f9500 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771e4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771e4340 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 --2- 172.21.15.106:0/2765297076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771e3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde627fc640 0 client.25637 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 -- 172.21.15.106:0/2765297076 >> 172.21.15.106:0/2765297076 conn(0x55e2771d8030 msgr2=0x55e2771d9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 -- 172.21.15.106:0/2765297076 shutdown_connections 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 1 -- 172.21.15.106:0/2765297076 wait complete. 2024-03-28T02:13:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.980+0000 7fde83fd0740 20 client.25637 trim_cache size 0 max 16384 2024-03-28T02:13:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.993+0000 7fde83fd0740 1 Processor -- start 2024-03-28T02:13:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.993+0000 7fde83fd0740 1 -- start start 2024-03-28T02:13:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771e6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771d6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde83fd0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1ac5b0 con 0x55e2771e6e00 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde83fd0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1adf30 con 0x55e2771de030 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde83fd0740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1acc10 con 0x55e2771dd710 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771f22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54212/0 (socket says 172.21.15.106:54212) 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7b7fe640 1 -- 172.21.15.106:0/3171397376 learned_addr learned my addr 172.21.15.106:0/3171397376 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde73fff640 1 --2- 172.21.15.106:0/3171397376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771d6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7affd640 1 --2- 172.21.15.106:0/3171397376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771e6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7b7fe640 1 -- 172.21.15.106:0/3171397376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 msgr2=0x55e2771e6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7b7fe640 1 --2- 172.21.15.106:0/3171397376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771e6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7b7fe640 1 -- 172.21.15.106:0/3171397376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 msgr2=0x55e2771d6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7b7fe640 1 --2- 172.21.15.106:0/3171397376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771d6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7b7fe640 1 -- 172.21.15.106:0/3171397376 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2771dde60 con 0x55e2771e6e00 2024-03-28T02:13:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde7b7fe640 1 --2- 172.21.15.106:0/3171397376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771f22c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fde74099850 tx=0x7fde74013ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268392acf61797d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.994+0000 7fde70ff9640 1 -- 172.21.15.106:0/3171397376 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74025040 con 0x55e2771e6e00 2024-03-28T02:13:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde70ff9640 1 -- 172.21.15.106:0/3171397376 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde7409ced0 con 0x55e2771e6e00 2024-03-28T02:13:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde70ff9640 1 -- 172.21.15.106:0/3171397376 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74077640 con 0x55e2771e6e00 2024-03-28T02:13:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 -- 172.21.15.106:0/3171397376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 msgr2=0x55e2771f22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 --2- 172.21.15.106:0/3171397376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771f22c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fde74099850 tx=0x7fde74013ee0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 -- 172.21.15.106:0/3171397376 shutdown_connections 2024-03-28T02:13:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 --2- 172.21.15.106:0/3171397376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771e6e00 0x55e2771f22c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 --2- 172.21.15.106:0/3171397376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771de030 0x55e2771d6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 --2- 172.21.15.106:0/3171397376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771dd710 0x55e2771e6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 -- 172.21.15.106:0/3171397376 >> 172.21.15.106:0/3171397376 conn(0x55e2771d8030 msgr2=0x55e2771d6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 -- 172.21.15.106:0/3171397376 shutdown_connections 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 -- 172.21.15.106:0/3171397376 wait complete. 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.995+0000 7fde83fd0740 1 -- start start 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde83fd0740 1 Processor -- start 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771dd710 0x55e2771eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771de030 0x55e2771eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde83fd0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 0x55e27724b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde83fd0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1ad8d0 con 0x55e2771dd710 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde83fd0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1ad270 con 0x55e2771e6e00 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde83fd0740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1aa770 con 0x55e2771de030 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde83fd0740 10 client.0 _ll_register_callbacks cb 0x7fde64004370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771dd710 0x55e2771eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771dd710 0x55e2771eaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:54218/0 (socket says 172.21.15.106:54218) 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde73fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771de030 0x55e2771eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7affd640 1 -- 172.21.15.106:0/1646868932 learned_addr learned my addr 172.21.15.106:0/1646868932 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7b7fe640 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 0x55e27724b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7affd640 1 -- 172.21.15.106:0/1646868932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771de030 msgr2=0x55e2771eaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7affd640 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771de030 0x55e2771eaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7affd640 1 -- 172.21.15.106:0/1646868932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 msgr2=0x55e27724b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7affd640 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 0x55e27724b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7affd640 1 -- 172.21.15.106:0/1646868932 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e27724bf90 con 0x55e2771dd710 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde7affd640 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771dd710 0x55e2771eaa30 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fde6c014040 tx=0x7fde6c009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c4c6e9b0aebbde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde627fc640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.996+0000 7fde627fc640 1 -- 172.21.15.106:0/1646868932 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c005190 con 0x55e2771dd710 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde627fc640 1 -- 172.21.15.106:0/1646868932 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fde6c005330 con 0x55e2771dd710 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde83fd0740 1 -- 172.21.15.106:0/1646868932 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde7409ced0 con 0x55e2771dd710 2024-03-28T02:13:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde627fc640 1 -- 172.21.15.106:0/1646868932 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde6c03d050 con 0x55e2771dd710 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde83fd0740 10 client.15789 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde83fd0740 1 -- 172.21.15.106:0/1646868932 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e277253320 con 0x55e2771dd710 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde627fc640 1 -- 172.21.15.106:0/1646868932 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde7409ced0 con 0x55e2771dd710 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde627fc640 1 -- 172.21.15.106:0/1646868932 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fde6c004710 con 0x55e2771dd710 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde83fd0740 10 client.15789 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde83fd0740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde83fd0740 20 client.15789 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde83fd0740 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e277253890 0x55e277255cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde83fd0740 4 client.15789 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde83fd0740 1 -- 172.21.15.106:0/1646868932 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e2771493a0 con 0x55e277253890 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.997+0000 7fde73fff640 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e277253890 0x55e277255cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.998+0000 7fde627fc640 10 client.15789 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:16.998+0000 7fde73fff640 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e277253890 0x55e277255cb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a26388e7dff9df1 server_cookie=3fb9a08368cc752d in_seq=0 out_seq=0 2024-03-28T02:13:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde627fc640 1 -- 172.21.15.106:0/1646868932 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e2771493a0 con 0x55e277253890 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde627fc640 10 client.15789 handle_command_reply: tid=0 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 client.15789 shutdown 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 -- 172.21.15.106:0/1646868932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771dd710 msgr2=0x55e2771eaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771dd710 0x55e2771eaa30 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fde6c014040 tx=0x7fde6c009c80 comp rx=0 tx=0).stop 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 -- 172.21.15.106:0/1646868932 shutdown_connections 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55e277253890 0x55e277255cb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e2771e6e00 0x55e27724b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e2771de030 0x55e2771eaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 --2- 172.21.15.106:0/1646868932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e2771dd710 0x55e2771eaa30 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde627fc640 0 client.15789 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 -- 172.21.15.106:0/1646868932 >> 172.21.15.106:0/1646868932 conn(0x55e2771d8030 msgr2=0x55e2771e4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 -- 172.21.15.106:0/1646868932 shutdown_connections 2024-03-28T02:13:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.006+0000 7fde83fd0740 1 -- 172.21.15.106:0/1646868932 wait complete. 2024-03-28T02:13:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde83fd0740 20 client.15789 trim_cache size 0 max 16384 2024-03-28T02:13:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:13:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:13:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:13:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 -- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fde680a2a90 msgr2=0x7fde680a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fde680a2a90 0x7fde680a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fde64006010 tx=0x7fde64005e40 comp rx=0 tx=0).stop 2024-03-28T02:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 -- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c105520 msgr2=0x7fde7c19b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c105520 0x7fde7c19b6a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fde7400eca0 tx=0x7fde7400b690 comp rx=0 tx=0).stop 2024-03-28T02:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 -- 172.21.15.106:0/846067279 shutdown_connections 2024-03-28T02:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fde680a2a90 0x7fde680a4f50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde7c105520 0x7fde7c19b6a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde7c101190 0x7fde7c197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 --2- 172.21.15.106:0/846067279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde7c1007f0 0x7fde7c196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 -- 172.21.15.106:0/846067279 >> 172.21.15.106:0/846067279 conn(0x7fde7c077370 msgr2=0x7fde7c0fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 -- 172.21.15.106:0/846067279 shutdown_connections 2024-03-28T02:13:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:17.007+0000 7fde81313640 1 -- 172.21.15.106:0/846067279 wait complete. 2024-03-28T02:13:17.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:17 smithi106 ceph-mon[31965]: pgmap v63: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 106 MiB/s wr, 61 op/s 2024-03-28T02:13:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:17 smithi178 ceph-mon[36521]: pgmap v63: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 106 MiB/s wr, 61 op/s 2024-03-28T02:13:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:17 smithi153 ceph-mon[36228]: pgmap v63: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 106 MiB/s wr, 61 op/s 2024-03-28T02:13:18.067 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:18.068 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:18.068 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:18.068 INFO:tasks.workunit.client.0.smithi106.stdout:iozone test complete. 2024-03-28T02:13:18.068 INFO:tasks.workunit.client.0.smithi106.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-28T02:13:18.134 INFO:tasks.workunit.client.0.smithi106.stdout: Iozone: Performance Test of File I/O 2024-03-28T02:13:18.134 INFO:tasks.workunit.client.0.smithi106.stdout: Version $Revision: 3.494 $ 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Compiled for 64 bit mode. 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Build: linux-AMD64 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-28T02:13:18.135 INFO:tasks.workunit.client.0.smithi106.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Run began: Thu Mar 28 02:13:18 2024 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Include close in write timing 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Include fsync in write timing 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: File size set to 10485760 kB 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Record Size 1024 kB 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Output is in kBytes/sec 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Time Resolution = 0.000001 seconds. 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Processor cache size set to 1024 kBytes. 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Processor cache line size set to 32 bytes. 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: File stride size set to 17 * record size. 2024-03-28T02:13:18.136 INFO:tasks.workunit.client.0.smithi106.stdout: Throughput test with 1 process 2024-03-28T02:13:18.137 INFO:tasks.workunit.client.0.smithi106.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-03-28T02:13:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:19 smithi106 ceph-mon[31965]: pgmap v64: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 106 MiB/s wr, 68 op/s 2024-03-28T02:13:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:19 smithi178 ceph-mon[36521]: pgmap v64: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 106 MiB/s wr, 68 op/s 2024-03-28T02:13:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:19 smithi153 ceph-mon[36228]: pgmap v64: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 106 MiB/s wr, 68 op/s 2024-03-28T02:13:21.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:21 smithi106 ceph-mon[31965]: pgmap v65: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-03-28T02:13:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:21 smithi178 ceph-mon[36521]: pgmap v65: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-03-28T02:13:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:21 smithi153 ceph-mon[36228]: pgmap v65: 129 pgs: 129 active+clean; 1.0 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-03-28T02:13:22.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:22 smithi178 ceph-mon[36521]: pgmap v66: 129 pgs: 129 active+clean; 3.1 GiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 267 MiB/s wr, 243 op/s 2024-03-28T02:13:22.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:22 smithi153 ceph-mon[36228]: pgmap v66: 129 pgs: 129 active+clean; 3.1 GiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 267 MiB/s wr, 243 op/s 2024-03-28T02:13:23.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:22 smithi106 ceph-mon[31965]: pgmap v66: 129 pgs: 129 active+clean; 3.1 GiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 267 MiB/s wr, 243 op/s 2024-03-28T02:13:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:23 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:13:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:23 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:13:24.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:23 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:13:24.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:24 smithi178 ceph-mon[36521]: pgmap v67: 129 pgs: 129 active+clean; 3.1 GiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 267 MiB/s wr, 239 op/s 2024-03-28T02:13:24.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:24 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:13:24.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:24 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:13:24.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:24 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:13:24.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:24 smithi153 ceph-mon[36228]: pgmap v67: 129 pgs: 129 active+clean; 3.1 GiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 267 MiB/s wr, 239 op/s 2024-03-28T02:13:24.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:24 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:13:24.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:24 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:13:24.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:24 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:13:25.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:24 smithi106 ceph-mon[31965]: pgmap v67: 129 pgs: 129 active+clean; 3.1 GiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 267 MiB/s wr, 239 op/s 2024-03-28T02:13:25.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:24 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:13:25.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:24 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:13:25.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:24 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:13:27.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:27 smithi106 ceph-mon[31965]: pgmap v68: 129 pgs: 129 active+clean; 3.1 GiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 267 MiB/s wr, 239 op/s 2024-03-28T02:13:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:27 smithi178 ceph-mon[36521]: pgmap v68: 129 pgs: 129 active+clean; 3.1 GiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 267 MiB/s wr, 239 op/s 2024-03-28T02:13:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:27 smithi153 ceph-mon[36228]: pgmap v68: 129 pgs: 129 active+clean; 3.1 GiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 267 MiB/s wr, 239 op/s 2024-03-28T02:13:29.273 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:29.273 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 initial writers = 944124.25 kB/sec 2024-03-28T02:13:29.273 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 initial writers = 944104.60 kB/sec 2024-03-28T02:13:29.273 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 944124.25 kB/sec 2024-03-28T02:13:29.273 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 944124.25 kB/sec 2024-03-28T02:13:29.273 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 944124.25 kB/sec 2024-03-28T02:13:29.274 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 10485760.00 kB 2024-03-28T02:13:29.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:29 smithi106 ceph-mon[31965]: pgmap v69: 129 pgs: 129 active+clean; 8.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 701 MiB/s wr, 430 op/s 2024-03-28T02:13:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:29 smithi178 ceph-mon[36521]: pgmap v69: 129 pgs: 129 active+clean; 8.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 701 MiB/s wr, 430 op/s 2024-03-28T02:13:29.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:29 smithi153 ceph-mon[36228]: pgmap v69: 129 pgs: 129 active+clean; 8.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 701 MiB/s wr, 430 op/s 2024-03-28T02:13:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.105+0000 7fb9289b3640 1 -- 172.21.15.106:0/1870025370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924074370 msgr2=0x7fb92410d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.105+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1870025370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924074370 0x7fb92410d9f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb918009900 tx=0x7fb91802bcd0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.105+0000 7fb9289b3640 1 -- 172.21.15.106:0/1870025370 shutdown_connections 2024-03-28T02:13:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.105+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1870025370 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb92410df30 0x7fb924110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.105+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1870025370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924074370 0x7fb92410d9f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.105+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1870025370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb924073a50 0x7fb924073e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.105+0000 7fb9289b3640 1 -- 172.21.15.106:0/1870025370 >> 172.21.15.106:0/1870025370 conn(0x7fb9240fd1a0 msgr2=0x7fb9240ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.105+0000 7fb9289b3640 1 -- 172.21.15.106:0/1870025370 shutdown_connections 2024-03-28T02:13:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.105+0000 7fb9289b3640 1 -- 172.21.15.106:0/1870025370 wait complete. 2024-03-28T02:13:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb9289b3640 1 Processor -- start 2024-03-28T02:13:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb9289b3640 1 -- start start 2024-03-28T02:13:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb9289b3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924073a50 0x7fb92419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb9289b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb924074370 0x7fb92419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb9289b3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb92410df30 0x7fb92419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb9289b3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb924113c50 con 0x7fb924073a50 2024-03-28T02:13:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb9289b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb924113ad0 con 0x7fb924074370 2024-03-28T02:13:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb9289b3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb924113dd0 con 0x7fb92410df30 2024-03-28T02:13:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb92259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924073a50 0x7fb92419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb92259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924073a50 0x7fb92419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:38926/0 (socket says 172.21.15.106:38926) 2024-03-28T02:13:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb92259c640 1 -- 172.21.15.106:0/1504367235 learned_addr learned my addr 172.21.15.106:0/1504367235 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb921d9b640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb924074370 0x7fb92419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb922d9d640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb92410df30 0x7fb92419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb92259c640 1 -- 172.21.15.106:0/1504367235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb92410df30 msgr2=0x7fb92419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb92259c640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb92410df30 0x7fb92419ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb92259c640 1 -- 172.21.15.106:0/1504367235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb924074370 msgr2=0x7fb92419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb92259c640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb924074370 0x7fb92419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.106+0000 7fb92259c640 1 -- 172.21.15.106:0/1504367235 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92419f150 con 0x7fb924073a50 2024-03-28T02:13:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.107+0000 7fb92259c640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924073a50 0x7fb92419a1b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00e9f0 tx=0x7fb91c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.107+0000 7fb9137fe640 1 -- 172.21.15.106:0/1504367235 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c00cd80 con 0x7fb924073a50 2024-03-28T02:13:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.107+0000 7fb9137fe640 1 -- 172.21.15.106:0/1504367235 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb91c010910 con 0x7fb924073a50 2024-03-28T02:13:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.107+0000 7fb9289b3640 1 -- 172.21.15.106:0/1504367235 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92419f440 con 0x7fb924073a50 2024-03-28T02:13:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.107+0000 7fb9137fe640 1 -- 172.21.15.106:0/1504367235 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c018990 con 0x7fb924073a50 2024-03-28T02:13:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.107+0000 7fb9289b3640 1 -- 172.21.15.106:0/1504367235 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9241a6cd0 con 0x7fb924073a50 2024-03-28T02:13:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.110+0000 7fb9137fe640 1 -- 172.21.15.106:0/1504367235 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fb91c010ac0 con 0x7fb924073a50 2024-03-28T02:13:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.110+0000 7fb92b598740 1 Processor -- start 2024-03-28T02:13:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.110+0000 7fb92b598740 1 -- start start 2024-03-28T02:13:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.110+0000 7fb92b598740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 0x5628c79dbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.110+0000 7fb92b598740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79dc030 0x5628c79e48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.110+0000 7fb92b598740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79e71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.110+0000 7fb92b598740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c793fb10 con 0x5628c79dc030 2024-03-28T02:13:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.110+0000 7fb92b598740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c78e8900 con 0x5628c79e4e00 2024-03-28T02:13:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.110+0000 7fb92b598740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c793bbb0 con 0x5628c79db710 2024-03-28T02:13:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb9137fe640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb90c0a2a10 0x7fb90c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb92259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79dc030 0x5628c79e48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb922d9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 0x5628c79dbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb921d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79e71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb92259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79dc030 0x5628c79e48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:38938/0 (socket says 172.21.15.106:38938) 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb922d9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 0x5628c79dbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:57218/0 (socket says 172.21.15.106:57218) 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb92259c640 1 -- 172.21.15.106:0/2801826940 learned_addr learned my addr 172.21.15.106:0/2801826940 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb921d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79e71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:53670/0 (socket says 172.21.15.106:53670) 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb922d9d640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb90c0a2a10 0x7fb90c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb9137fe640 1 -- 172.21.15.106:0/1504367235 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fb91c0cf480 con 0x7fb924073a50 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb92259c640 1 -- 172.21.15.106:0/2801826940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 msgr2=0x5628c79dbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb92259c640 1 --2- 172.21.15.106:0/2801826940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 0x5628c79dbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb92259c640 1 -- 172.21.15.106:0/2801826940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 msgr2=0x5628c79e71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb92259c640 1 --2- 172.21.15.106:0/2801826940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79e71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb92259c640 1 -- 172.21.15.106:0/2801826940 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628c7654210 con 0x5628c79dc030 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb921d9b640 1 --2- 172.21.15.106:0/2801826940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79e71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb92259c640 1 --2- 172.21.15.106:0/2801826940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79dc030 0x5628c79e48c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0a3a70 tx=0x7fb91c018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f16ac16f6c40a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb922d9d640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb90c0a2a10 0x7fb90c0a4ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb914009fd0 tx=0x7fb914004ab0 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb910ff9640 1 -- 172.21.15.106:0/2801826940 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c099bb0 con 0x5628c79dc030 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb910ff9640 1 -- 172.21.15.106:0/2801826940 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb91c0ccb70 con 0x5628c79dc030 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.111+0000 7fb910ff9640 1 -- 172.21.15.106:0/2801826940 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c0860c0 con 0x5628c79dc030 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.112+0000 7fb92b598740 1 -- 172.21.15.106:0/2801826940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79dc030 msgr2=0x5628c79e48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.112+0000 7fb92b598740 1 --2- 172.21.15.106:0/2801826940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79dc030 0x5628c79e48c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0a3a70 tx=0x7fb91c018420 comp rx=0 tx=0).stop 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.112+0000 7fb92b598740 1 -- 172.21.15.106:0/2801826940 shutdown_connections 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.112+0000 7fb92b598740 1 --2- 172.21.15.106:0/2801826940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79e71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.112+0000 7fb92b598740 1 --2- 172.21.15.106:0/2801826940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79dc030 0x5628c79e48c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.112+0000 7fb92b598740 1 --2- 172.21.15.106:0/2801826940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 0x5628c79dbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.112+0000 7fb92b598740 1 -- 172.21.15.106:0/2801826940 >> 172.21.15.106:0/2801826940 conn(0x5628c79d6030 msgr2=0x5628c79d6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.112+0000 7fb92b598740 1 -- 172.21.15.106:0/2801826940 shutdown_connections 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.112+0000 7fb92b598740 1 -- 172.21.15.106:0/2801826940 wait complete. 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92b598740 1 -- start start 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92b598740 1 Processor -- start 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92b598740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92b598740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92b598740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79e4e00 0x5628c79e28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92b598740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c79e7970 con 0x5628c79db710 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92b598740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c78fbd50 con 0x5628c79dc030 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92b598740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c79e7af0 con 0x5628c79e4e00 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92b598740 10 client.0 _ll_register_callbacks cb 0x5628c78fb720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:38948/0 (socket says 172.21.15.106:38948) 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb921d9b640 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92259c640 1 -- 172.21.15.106:0/3005068783 learned_addr learned my addr 172.21.15.106:0/3005068783 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb922d9d640 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79e4e00 0x5628c79e28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92259c640 1 -- 172.21.15.106:0/3005068783 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79e4e00 msgr2=0x5628c79e28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92259c640 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79e4e00 0x5628c79e28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92259c640 1 -- 172.21.15.106:0/3005068783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 msgr2=0x5628c79e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92259c640 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb92259c640 1 -- 172.21.15.106:0/3005068783 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628c7720620 con 0x5628c79db710 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.113+0000 7fb921d9b640 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92259c640 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e1e40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb91c09b840 tx=0x7fb91c099dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee76b65daddb29e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb909ffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb909ffb640 1 -- 172.21.15.106:0/3005068783 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c0cf190 con 0x5628c79db710 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb909ffb640 1 -- 172.21.15.106:0/3005068783 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fb91c0afbe0 con 0x5628c79db710 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92b598740 1 -- 172.21.15.106:0/3005068783 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb91c0ccb70 con 0x5628c79db710 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb909ffb640 1 -- 172.21.15.106:0/3005068783 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb91c0ba0d0 con 0x5628c79db710 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92b598740 10 client.15801 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92b598740 1 -- 172.21.15.106:0/3005068783 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5628c79f3bf0 con 0x5628c79db710 2024-03-28T02:13:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb909ffb640 1 -- 172.21.15.106:0/3005068783 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb91c0ccb70 con 0x5628c79db710 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb909ffb640 1 -- 172.21.15.106:0/3005068783 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb91c097160 con 0x5628c79db710 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92b598740 10 client.15801 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92b598740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92b598740 20 client.15801 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92b598740 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5628c79f4f40 0x5628c79f7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92b598740 4 client.15801 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.114+0000 7fb92b598740 1 -- 172.21.15.106:0/3005068783 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628c794d050 con 0x5628c79f4f40 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.115+0000 7fb921d9b640 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5628c79f4f40 0x5628c79f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.115+0000 7fb921d9b640 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5628c79f4f40 0x5628c79f7330 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60da1ab01ab57277 server_cookie=1b10402423859577 in_seq=0 out_seq=0 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.115+0000 7fb909ffb640 10 client.15801 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb909ffb640 1 -- 172.21.15.106:0/3005068783 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5628c794d050 con 0x5628c79f4f40 2024-03-28T02:13:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb909ffb640 10 client.15801 handle_command_reply: tid=0 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 client.15801 shutdown 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 -- 172.21.15.106:0/3005068783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 msgr2=0x5628c79e1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e1e40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb91c09b840 tx=0x7fb91c099dc0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 -- 172.21.15.106:0/3005068783 shutdown_connections 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5628c79f4f40 0x5628c79f7330 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79e4e00 0x5628c79e28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb909ffb640 0 client.15801 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 --2- 172.21.15.106:0/3005068783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e1e40 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 -- 172.21.15.106:0/3005068783 >> 172.21.15.106:0/3005068783 conn(0x5628c79d6030 msgr2=0x5628c79d7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 -- 172.21.15.106:0/3005068783 shutdown_connections 2024-03-28T02:13:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 1 -- 172.21.15.106:0/3005068783 wait complete. 2024-03-28T02:13:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.116+0000 7fb92b598740 20 client.15801 trim_cache size 0 max 16384 2024-03-28T02:13:30.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92b598740 1 Processor -- start 2024-03-28T02:13:30.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92b598740 1 -- start start 2024-03-28T02:13:30.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92b598740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92b598740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79dc030 0x5628c79f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92b598740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79f0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92b598740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241b0ab0 con 0x5628c79db710 2024-03-28T02:13:30.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92b598740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241b2430 con 0x5628c79e4e00 2024-03-28T02:13:30.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92b598740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241b1110 con 0x5628c79dc030 2024-03-28T02:13:30.128 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:38956/0 (socket says 172.21.15.106:38956) 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb92259c640 1 -- 172.21.15.106:0/1567953894 learned_addr learned my addr 172.21.15.106:0/1567953894 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb921d9b640 1 --2- 172.21.15.106:0/1567953894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79dc030 0x5628c79f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.130+0000 7fb922d9d640 1 --2- 172.21.15.106:0/1567953894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.131+0000 7fb92259c640 1 -- 172.21.15.106:0/1567953894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79dc030 msgr2=0x5628c79f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.131+0000 7fb92259c640 1 --2- 172.21.15.106:0/1567953894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79dc030 0x5628c79f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.131+0000 7fb92259c640 1 -- 172.21.15.106:0/1567953894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 msgr2=0x5628c79f0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.131+0000 7fb92259c640 1 --2- 172.21.15.106:0/1567953894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79f0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.131+0000 7fb92259c640 1 -- 172.21.15.106:0/1567953894 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628c79db540 con 0x5628c79db710 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.131+0000 7fb92259c640 1 --2- 172.21.15.106:0/1567953894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e4570 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb91c098280 tx=0x7fb91c09fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9598ec2e80a282 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.131+0000 7fb90bfff640 1 -- 172.21.15.106:0/1567953894 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c09b680 con 0x5628c79db710 2024-03-28T02:13:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.131+0000 7fb90bfff640 1 -- 172.21.15.106:0/1567953894 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb91c0b4070 con 0x5628c79db710 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.131+0000 7fb90bfff640 1 -- 172.21.15.106:0/1567953894 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c097b50 con 0x5628c79db710 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 -- 172.21.15.106:0/1567953894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 msgr2=0x5628c79e4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 --2- 172.21.15.106:0/1567953894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e4570 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb91c098280 tx=0x7fb91c09fad0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 -- 172.21.15.106:0/1567953894 shutdown_connections 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 --2- 172.21.15.106:0/1567953894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79e4e00 0x5628c79f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 --2- 172.21.15.106:0/1567953894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79dc030 0x5628c79f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 --2- 172.21.15.106:0/1567953894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79db710 0x5628c79e4570 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 -- 172.21.15.106:0/1567953894 >> 172.21.15.106:0/1567953894 conn(0x5628c79d6030 msgr2=0x5628c79dd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 -- 172.21.15.106:0/1567953894 shutdown_connections 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 -- 172.21.15.106:0/1567953894 wait complete. 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 -- start start 2024-03-28T02:13:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 Processor -- start 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 0x5628c79e8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79e4e00 0x5628c7a4f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241b1dd0 con 0x5628c79e4e00 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241b1770 con 0x5628c79dc030 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241aec70 con 0x5628c79db710 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92b598740 10 client.0 _ll_register_callbacks cb 0x7fb91c009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb922d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79e4e00 0x5628c7a4f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb921d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb92259c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 0x5628c79e8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb922d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79e4e00 0x5628c7a4f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:38968/0 (socket says 172.21.15.106:38968) 2024-03-28T02:13:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.132+0000 7fb922d9d640 1 -- 172.21.15.106:0/4141952960 learned_addr learned my addr 172.21.15.106:0/4141952960 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb922d9d640 1 -- 172.21.15.106:0/4141952960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 msgr2=0x5628c79e8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb922d9d640 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 0x5628c79e8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb922d9d640 1 -- 172.21.15.106:0/4141952960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 msgr2=0x5628c79e8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb922d9d640 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb922d9d640 1 -- 172.21.15.106:0/4141952960 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628c7a4fab0 con 0x5628c79e4e00 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb921d9b640 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb922d9d640 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79e4e00 0x5628c7a4f4c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb914005fd0 tx=0x7fb914004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89640aa1e896fda server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb909ffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb909ffb640 1 -- 172.21.15.106:0/4141952960 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91400fcc0 con 0x5628c79e4e00 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb909ffb640 1 -- 172.21.15.106:0/4141952960 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fb914004540 con 0x5628c79e4e00 2024-03-28T02:13:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb92b598740 1 -- 172.21.15.106:0/4141952960 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb91c0b4070 con 0x5628c79e4e00 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb909ffb640 1 -- 172.21.15.106:0/4141952960 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb914035050 con 0x5628c79e4e00 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb92b598740 10 client.15807 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb92b598740 1 -- 172.21.15.106:0/4141952960 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5628c7a50da0 con 0x5628c79e4e00 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb909ffb640 1 -- 172.21.15.106:0/4141952960 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb91c0b4070 con 0x5628c79e4e00 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb909ffb640 1 -- 172.21.15.106:0/4141952960 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb914005330 con 0x5628c79e4e00 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb92b598740 10 client.15807 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb92b598740 10 client.15807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.133+0000 7fb92b598740 20 client.15807 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.134+0000 7fb92b598740 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5628c7a51500 0x5628c7a53920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.134+0000 7fb92b598740 4 client.15807 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.134+0000 7fb92b598740 1 -- 172.21.15.106:0/4141952960 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5628c79473a0 con 0x5628c7a51500 2024-03-28T02:13:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.134+0000 7fb92259c640 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5628c7a51500 0x5628c7a53920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.134+0000 7fb909ffb640 10 client.15807 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.134+0000 7fb92259c640 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5628c7a51500 0x5628c7a53920 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=519f6b292bec4f2c server_cookie=aac5413b4d68c869 in_seq=0 out_seq=0 2024-03-28T02:13:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb909ffb640 1 -- 172.21.15.106:0/4141952960 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5628c79473a0 con 0x5628c7a51500 2024-03-28T02:13:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb909ffb640 10 client.15807 handle_command_reply: tid=0 2024-03-28T02:13:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 client.15807 shutdown 2024-03-28T02:13:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 -- 172.21.15.106:0/4141952960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79e4e00 msgr2=0x5628c7a4f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79e4e00 0x5628c7a4f4c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb914005fd0 tx=0x7fb914004a40 comp rx=0 tx=0).stop 2024-03-28T02:13:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 -- 172.21.15.106:0/4141952960 shutdown_connections 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x5628c7a51500 0x5628c7a53920 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5628c79e4e00 0x5628c7a4f4c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5628c79dc030 0x5628c79e8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 --2- 172.21.15.106:0/4141952960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5628c79db710 0x5628c79e8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb909ffb640 0 client.15807 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 -- 172.21.15.106:0/4141952960 >> 172.21.15.106:0/4141952960 conn(0x5628c79d6030 msgr2=0x5628c79e1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 -- 172.21.15.106:0/4141952960 shutdown_connections 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 1 -- 172.21.15.106:0/4141952960 wait complete. 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb92b598740 20 client.15807 trim_cache size 0 max 16384 2024-03-28T02:13:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:13:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:13:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:13:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:13:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb9289b3640 1 -- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb90c0a2a10 msgr2=0x7fb90c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb90c0a2a10 0x7fb90c0a4ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb914009fd0 tx=0x7fb914004ab0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb9289b3640 1 -- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924073a50 msgr2=0x7fb92419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.146+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924073a50 0x7fb92419a1b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00e9f0 tx=0x7fb91c00eeb0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.147+0000 7fb9289b3640 1 -- 172.21.15.106:0/1504367235 shutdown_connections 2024-03-28T02:13:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.147+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb90c0a2a10 0x7fb90c0a4ed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.147+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb92410df30 0x7fb92419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.147+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb924074370 0x7fb92419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.147+0000 7fb9289b3640 1 --2- 172.21.15.106:0/1504367235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb924073a50 0x7fb92419a1b0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.147+0000 7fb9289b3640 1 -- 172.21.15.106:0/1504367235 >> 172.21.15.106:0/1504367235 conn(0x7fb9240fd1a0 msgr2=0x7fb9240fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.147+0000 7fb9289b3640 1 -- 172.21.15.106:0/1504367235 shutdown_connections 2024-03-28T02:13:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:30.147+0000 7fb9289b3640 1 -- 172.21.15.106:0/1504367235 wait complete. 2024-03-28T02:13:31.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:31 smithi106 ceph-mon[31965]: pgmap v70: 129 pgs: 129 active+clean; 8.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 701 MiB/s wr, 422 op/s 2024-03-28T02:13:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:31 smithi178 ceph-mon[36521]: pgmap v70: 129 pgs: 129 active+clean; 8.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 701 MiB/s wr, 422 op/s 2024-03-28T02:13:31.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:31 smithi153 ceph-mon[36228]: pgmap v70: 129 pgs: 129 active+clean; 8.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 701 MiB/s wr, 422 op/s 2024-03-28T02:13:33.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:33 smithi106 ceph-mon[31965]: pgmap v71: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 854 MiB/s wr, 467 op/s 2024-03-28T02:13:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:33 smithi178 ceph-mon[36521]: pgmap v71: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 854 MiB/s wr, 467 op/s 2024-03-28T02:13:33.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:33 smithi153 ceph-mon[36228]: pgmap v71: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 854 MiB/s wr, 467 op/s 2024-03-28T02:13:35.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:35 smithi106 ceph-mon[31965]: pgmap v72: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 586 MiB/s wr, 236 op/s 2024-03-28T02:13:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:35 smithi178 ceph-mon[36521]: pgmap v72: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 586 MiB/s wr, 236 op/s 2024-03-28T02:13:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:35 smithi153 ceph-mon[36228]: pgmap v72: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 586 MiB/s wr, 236 op/s 2024-03-28T02:13:37.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:37 smithi106 ceph-mon[31965]: pgmap v73: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 586 MiB/s wr, 236 op/s 2024-03-28T02:13:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:37 smithi178 ceph-mon[36521]: pgmap v73: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 586 MiB/s wr, 236 op/s 2024-03-28T02:13:37.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:37 smithi153 ceph-mon[36228]: pgmap v73: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 586 MiB/s wr, 236 op/s 2024-03-28T02:13:39.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:39 smithi106 ceph-mon[31965]: pgmap v74: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 943 MiB/s wr, 430 op/s 2024-03-28T02:13:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:39 smithi178 ceph-mon[36521]: pgmap v74: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 943 MiB/s wr, 430 op/s 2024-03-28T02:13:39.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:39 smithi153 ceph-mon[36228]: pgmap v74: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 943 MiB/s wr, 430 op/s 2024-03-28T02:13:41.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:41 smithi106 ceph-mon[31965]: pgmap v75: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 509 MiB/s wr, 240 op/s 2024-03-28T02:13:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:41 smithi178 ceph-mon[36521]: pgmap v75: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 509 MiB/s wr, 240 op/s 2024-03-28T02:13:41.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:41 smithi153 ceph-mon[36228]: pgmap v75: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 509 MiB/s wr, 240 op/s 2024-03-28T02:13:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.290+0000 7fb8a0377640 1 -- 172.21.15.106:0/3131097930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89810fde0 msgr2=0x7fb89810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.290+0000 7fb8a0377640 1 --2- 172.21.15.106:0/3131097930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89810fde0 0x7fb89810da00 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb888009900 tx=0x7fb88802bc50 comp rx=0 tx=0).stop 2024-03-28T02:13:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.291+0000 7fb8a0377640 1 -- 172.21.15.106:0/3131097930 shutdown_connections 2024-03-28T02:13:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.291+0000 7fb8a0377640 1 --2- 172.21.15.106:0/3131097930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb89811ae80 0x7fb89811d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.291+0000 7fb8a0377640 1 --2- 172.21.15.106:0/3131097930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb898110250 0x7fb89810df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.291+0000 7fb8a0377640 1 --2- 172.21.15.106:0/3131097930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89810fde0 0x7fb89810da00 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.291+0000 7fb8a0377640 1 -- 172.21.15.106:0/3131097930 >> 172.21.15.106:0/3131097930 conn(0x7fb89806c740 msgr2=0x7fb89806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.291+0000 7fb8a0377640 1 -- 172.21.15.106:0/3131097930 shutdown_connections 2024-03-28T02:13:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.292+0000 7fb8a0377640 1 -- 172.21.15.106:0/3131097930 wait complete. 2024-03-28T02:13:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.292+0000 7fb8a0377640 1 Processor -- start 2024-03-28T02:13:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.292+0000 7fb8a0377640 1 -- start start 2024-03-28T02:13:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.292+0000 7fb8a0377640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89810fde0 0x7fb8981a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.292+0000 7fb8a0377640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb898110250 0x7fb8981a72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.292+0000 7fb8a0377640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89811ae80 0x7fb8981ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.292+0000 7fb8a0377640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898120910 con 0x7fb89811ae80 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.292+0000 7fb8a0377640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898120790 con 0x7fb89810fde0 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.292+0000 7fb8a0377640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898120a90 con 0x7fb898110250 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e914640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89811ae80 0x7fb8981ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e914640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89811ae80 0x7fb8981ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40008/0 (socket says 172.21.15.106:40008) 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e914640 1 -- 172.21.15.106:0/380123969 learned_addr learned my addr 172.21.15.106:0/380123969 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e113640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89810fde0 0x7fb8981a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89d912640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb898110250 0x7fb8981a72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e914640 1 -- 172.21.15.106:0/380123969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb898110250 msgr2=0x7fb8981a72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e914640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb898110250 0x7fb8981a72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e914640 1 -- 172.21.15.106:0/380123969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89810fde0 msgr2=0x7fb8981a6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e914640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89810fde0 0x7fb8981a6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e914640 1 -- 172.21.15.106:0/380123969 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8981abda0 con 0x7fb89811ae80 2024-03-28T02:13:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb89e914640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89811ae80 0x7fb8981ab670 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb89000b9e0 tx=0x7fb89000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb88f7fe640 1 -- 172.21.15.106:0/380123969 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb890019070 con 0x7fb89811ae80 2024-03-28T02:13:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.293+0000 7fb8a0377640 1 -- 172.21.15.106:0/380123969 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8981ac090 con 0x7fb89811ae80 2024-03-28T02:13:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.294+0000 7fb88f7fe640 1 -- 172.21.15.106:0/380123969 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb890014440 con 0x7fb89811ae80 2024-03-28T02:13:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.294+0000 7fb8a0377640 1 -- 172.21.15.106:0/380123969 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8981b3920 con 0x7fb89811ae80 2024-03-28T02:13:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.294+0000 7fb88f7fe640 1 -- 172.21.15.106:0/380123969 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8900125c0 con 0x7fb89811ae80 2024-03-28T02:13:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.296+0000 7fb88f7fe640 1 -- 172.21.15.106:0/380123969 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fb89001c070 con 0x7fb89811ae80 2024-03-28T02:13:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.297+0000 7fb8a303a740 1 Processor -- start 2024-03-28T02:13:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.297+0000 7fb8a303a740 1 -- start start 2024-03-28T02:13:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb88f7fe640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb8840a2a90 0x7fb8840a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df16710 0x55f25df16af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 0x55f25df1f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df1fe00 0x55f25df221f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb8a303a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25de7ab10 con 0x55f25df16710 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb8a303a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25de23900 con 0x55f25df17030 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb8a303a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25de76bb0 con 0x55f25df1fe00 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89d912640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb8840a2a90 0x7fb8840a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89e113640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df16710 0x55f25df16af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb88f7fe640 1 -- 172.21.15.106:0/380123969 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fb8900ce5a0 con 0x7fb89811ae80 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89e113640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df16710 0x55f25df16af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40014/0 (socket says 172.21.15.106:40014) 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89e113640 1 -- 172.21.15.106:0/2385526520 learned_addr learned my addr 172.21.15.106:0/2385526520 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89d912640 1 --2- 172.21.15.106:0/2385526520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 0x55f25df1f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89e113640 1 -- 172.21.15.106:0/2385526520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df1fe00 msgr2=0x55f25df221f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89e113640 1 --2- 172.21.15.106:0/2385526520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df1fe00 0x55f25df221f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89e113640 1 -- 172.21.15.106:0/2385526520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 msgr2=0x55f25df1f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89e113640 1 --2- 172.21.15.106:0/2385526520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 0x55f25df1f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89e113640 1 -- 172.21.15.106:0/2385526520 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f25db8f210 con 0x55f25df16710 2024-03-28T02:13:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.298+0000 7fb89e113640 1 --2- 172.21.15.106:0/2385526520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df16710 0x55f25df16af0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb888005ab0 tx=0x7fb888005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f6ec8adc6bdd77 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.299+0000 7fb89d912640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb8840a2a90 0x7fb8840a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb8980731c0 tx=0x7fb89400bb30 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.299+0000 7fb88cbfe640 1 -- 172.21.15.106:0/2385526520 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888042070 con 0x55f25df16710 2024-03-28T02:13:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.299+0000 7fb88cbfe640 1 -- 172.21.15.106:0/2385526520 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb888004240 con 0x55f25df16710 2024-03-28T02:13:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.299+0000 7fb88cbfe640 1 -- 172.21.15.106:0/2385526520 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888004f10 con 0x55f25df16710 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.300+0000 7fb8a303a740 1 -- 172.21.15.106:0/2385526520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df16710 msgr2=0x55f25df16af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.300+0000 7fb8a303a740 1 --2- 172.21.15.106:0/2385526520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df16710 0x55f25df16af0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb888005ab0 tx=0x7fb888005e80 comp rx=0 tx=0).stop 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.300+0000 7fb8a303a740 1 -- 172.21.15.106:0/2385526520 shutdown_connections 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.300+0000 7fb8a303a740 1 --2- 172.21.15.106:0/2385526520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df1fe00 0x55f25df221f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.300+0000 7fb8a303a740 1 --2- 172.21.15.106:0/2385526520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 0x55f25df1f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.300+0000 7fb8a303a740 1 --2- 172.21.15.106:0/2385526520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df16710 0x55f25df16af0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.300+0000 7fb8a303a740 1 -- 172.21.15.106:0/2385526520 >> 172.21.15.106:0/2385526520 conn(0x55f25df11030 msgr2=0x55f25df11410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.300+0000 7fb8a303a740 1 -- 172.21.15.106:0/2385526520 shutdown_connections 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.300+0000 7fb8a303a740 1 -- 172.21.15.106:0/2385526520 wait complete. 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.301+0000 7fb8a303a740 1 -- start start 2024-03-28T02:13:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.301+0000 7fb8a303a740 1 Processor -- start 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df16710 0x55f25df1cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df17030 0x55f25df1d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb8a303a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25df229d0 con 0x55f25df1fe00 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb8a303a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25de36d50 con 0x55f25df16710 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb8a303a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25df22b50 con 0x55f25df17030 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb8a303a740 10 client.0 _ll_register_callbacks cb 0x55f25de36720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb89e113640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df16710 0x55f25df1cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb89e113640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df16710 0x55f25df1cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:51884/0 (socket says 172.21.15.106:51884) 2024-03-28T02:13:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb89e113640 1 -- 172.21.15.106:0/3039369138 learned_addr learned my addr 172.21.15.106:0/3039369138 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.302+0000 7fb89e914640 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.303+0000 7fb89e113640 1 -- 172.21.15.106:0/3039369138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df17030 msgr2=0x55f25df1d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.303+0000 7fb89e113640 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df17030 0x55f25df1d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.303+0000 7fb89e113640 1 -- 172.21.15.106:0/3039369138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 msgr2=0x55f25df1d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.303+0000 7fb89e113640 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.303+0000 7fb89e113640 1 -- 172.21.15.106:0/3039369138 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f25dc5b620 con 0x55f25df16710 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.303+0000 7fb89e914640 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.303+0000 7fb89e113640 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df16710 0x55f25df1cea0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb888009850 tx=0x7fb88802bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=614bffab83ce561f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.304+0000 7fb881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.304+0000 7fb881ffb640 1 -- 172.21.15.106:0/3039369138 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88804f060 con 0x55f25df16710 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.304+0000 7fb881ffb640 1 -- 172.21.15.106:0/3039369138 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fb888004f10 con 0x55f25df16710 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.304+0000 7fb8a303a740 1 -- 172.21.15.106:0/3039369138 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb888004240 con 0x55f25df16710 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.304+0000 7fb881ffb640 1 -- 172.21.15.106:0/3039369138 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb888056050 con 0x55f25df16710 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.304+0000 7fb8a303a740 10 client.25673 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.304+0000 7fb8a303a740 1 -- 172.21.15.106:0/3039369138 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f25df2ec90 con 0x55f25df16710 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.304+0000 7fb881ffb640 1 -- 172.21.15.106:0/3039369138 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb888004240 con 0x55f25df16710 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.305+0000 7fb881ffb640 1 -- 172.21.15.106:0/3039369138 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb888047a30 con 0x55f25df16710 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.305+0000 7fb8a303a740 10 client.25673 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.305+0000 7fb8a303a740 10 client.25673 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.305+0000 7fb8a303a740 20 client.25673 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.305+0000 7fb8a303a740 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f25df2ffe0 0x55f25df323d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.305+0000 7fb8a303a740 4 client.25673 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:13:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.305+0000 7fb8a303a740 1 -- 172.21.15.106:0/3039369138 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f25de88050 con 0x55f25df2ffe0 2024-03-28T02:13:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.305+0000 7fb89d912640 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f25df2ffe0 0x55f25df323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.306+0000 7fb89d912640 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f25df2ffe0 0x55f25df323d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=321e78c9149e7bcf server_cookie=e1c9ff4ba20aa4d in_seq=0 out_seq=0 2024-03-28T02:13:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.306+0000 7fb881ffb640 10 client.25673 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.307+0000 7fb881ffb640 1 -- 172.21.15.106:0/3039369138 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f25de88050 con 0x55f25df2ffe0 2024-03-28T02:13:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.307+0000 7fb881ffb640 10 client.25673 handle_command_reply: tid=0 2024-03-28T02:13:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.308+0000 7fb8a303a740 1 client.25673 shutdown 2024-03-28T02:13:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 -- 172.21.15.106:0/3039369138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df16710 msgr2=0x55f25df1cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df16710 0x55f25df1cea0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb888009850 tx=0x7fb88802bc50 comp rx=0 tx=0).stop 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 -- 172.21.15.106:0/3039369138 shutdown_connections 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f25df2ffe0 0x55f25df323d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df17030 0x55f25df1d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 --2- 172.21.15.106:0/3039369138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df16710 0x55f25df1cea0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 -- 172.21.15.106:0/3039369138 >> 172.21.15.106:0/3039369138 conn(0x55f25df11030 msgr2=0x55f25df12d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb881ffb640 0 client.25673 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 -- 172.21.15.106:0/3039369138 shutdown_connections 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 1 -- 172.21.15.106:0/3039369138 wait complete. 2024-03-28T02:13:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.309+0000 7fb8a303a740 20 client.25673 trim_cache size 0 max 16384 2024-03-28T02:13:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb8a303a740 1 Processor -- start 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb8a303a740 1 -- start start 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df16710 0x55f25df1f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 0x55f25df1d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb8a303a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981bd6d0 con 0x55f25df1fe00 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb8a303a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981bf050 con 0x55f25df17030 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb8a303a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981bdd30 con 0x55f25df16710 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb89e914640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb89e914640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40030/0 (socket says 172.21.15.106:40030) 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb89e914640 1 -- 172.21.15.106:0/1071364892 learned_addr learned my addr 172.21.15.106:0/1071364892 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb89d912640 1 --2- 172.21.15.106:0/1071364892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 0x55f25df1d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb89e914640 1 -- 172.21.15.106:0/1071364892 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df16710 msgr2=0x55f25df1f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb89e914640 1 --2- 172.21.15.106:0/1071364892 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df16710 0x55f25df1f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb89e914640 1 -- 172.21.15.106:0/1071364892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 msgr2=0x55f25df1d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb89e914640 1 --2- 172.21.15.106:0/1071364892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 0x55f25df1d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.328+0000 7fb89e914640 1 -- 172.21.15.106:0/1071364892 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f25df16e60 con 0x55f25df1fe00 2024-03-28T02:13:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.329+0000 7fb89e914640 1 --2- 172.21.15.106:0/1071364892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1a100 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb8900cbf70 tx=0x7fb890096ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db231cfe500884c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.329+0000 7fb883fff640 1 -- 172.21.15.106:0/1071364892 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8900b3070 con 0x55f25df1fe00 2024-03-28T02:13:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.329+0000 7fb883fff640 1 -- 172.21.15.106:0/1071364892 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8900a3130 con 0x55f25df1fe00 2024-03-28T02:13:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.329+0000 7fb883fff640 1 -- 172.21.15.106:0/1071364892 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb890097240 con 0x55f25df1fe00 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.330+0000 7fb8a303a740 1 -- 172.21.15.106:0/1071364892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 msgr2=0x55f25df1a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.330+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1071364892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1a100 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb8900cbf70 tx=0x7fb890096ce0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.330+0000 7fb8a303a740 1 -- 172.21.15.106:0/1071364892 shutdown_connections 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.330+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1071364892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df1fe00 0x55f25df1a100 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.330+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1071364892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df17030 0x55f25df1d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.330+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1071364892 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df16710 0x55f25df1f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.330+0000 7fb8a303a740 1 -- 172.21.15.106:0/1071364892 >> 172.21.15.106:0/1071364892 conn(0x55f25df11030 msgr2=0x55f25df176c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.330+0000 7fb8a303a740 1 -- 172.21.15.106:0/1071364892 shutdown_connections 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.330+0000 7fb8a303a740 1 -- 172.21.15.106:0/1071364892 wait complete. 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb8a303a740 1 -- start start 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb8a303a740 1 Processor -- start 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df16710 0x55f25df32ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df17030 0x55f25df33010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb8a303a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df1fe00 0x55f25df33570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb8a303a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981be9f0 con 0x55f25df17030 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb8a303a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981be390 con 0x55f25df1fe00 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb8a303a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981bb890 con 0x55f25df16710 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb89d912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df17030 0x55f25df33010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb8a303a740 10 client.0 _ll_register_callbacks cb 0x7fb888004c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb89d912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df17030 0x55f25df33010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:40036/0 (socket says 172.21.15.106:40036) 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.331+0000 7fb89d912640 1 -- 172.21.15.106:0/1877920865 learned_addr learned my addr 172.21.15.106:0/1877920865 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.332+0000 7fb89e914640 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df1fe00 0x55f25df33570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.332+0000 7fb89d912640 1 -- 172.21.15.106:0/1877920865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df16710 msgr2=0x55f25df32ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.332+0000 7fb89d912640 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df16710 0x55f25df32ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.332+0000 7fb89d912640 1 -- 172.21.15.106:0/1877920865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df1fe00 msgr2=0x55f25df33570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.332+0000 7fb89d912640 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df1fe00 0x55f25df33570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.332+0000 7fb89d912640 1 -- 172.21.15.106:0/1877920865 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f25df8a7e0 con 0x55f25df17030 2024-03-28T02:13:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.332+0000 7fb89e914640 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df1fe00 0x55f25df33570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:13:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.332+0000 7fb89d912640 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df17030 0x55f25df33010 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb89400b930 tx=0x7fb89401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38642b288f0af059 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.332+0000 7fb881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-28T02:13:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.333+0000 7fb881ffb640 1 -- 172.21.15.106:0/1877920865 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89402d070 con 0x55f25df17030 2024-03-28T02:13:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.333+0000 7fb881ffb640 1 -- 172.21.15.106:0/1877920865 <== mon.0 v2:172.21.15.106:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7fb894012840 con 0x55f25df17030 2024-03-28T02:13:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.333+0000 7fb8a303a740 1 -- 172.21.15.106:0/1877920865 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb8900a3130 con 0x55f25df17030 2024-03-28T02:13:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.333+0000 7fb881ffb640 1 -- 172.21.15.106:0/1877920865 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb894037050 con 0x55f25df17030 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.333+0000 7fb8a303a740 10 client.15825 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.333+0000 7fb8a303a740 1 -- 172.21.15.106:0/1877920865 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f25df8bb50 con 0x55f25df17030 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.334+0000 7fb881ffb640 1 -- 172.21.15.106:0/1877920865 <== mon.0 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8900a3130 con 0x55f25df17030 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.334+0000 7fb881ffb640 1 -- 172.21.15.106:0/1877920865 <== mon.0 v2:172.21.15.106:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb894029870 con 0x55f25df17030 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.334+0000 7fb8a303a740 10 client.15825 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.334+0000 7fb8a303a740 10 client.15825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.334+0000 7fb8a303a740 20 client.15825 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.334+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f25df8c1d0 0x55f25df8e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.334+0000 7fb8a303a740 4 client.15825 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.334+0000 7fb8a303a740 1 -- 172.21.15.106:0/1877920865 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f25de823a0 con 0x55f25df8c1d0 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.334+0000 7fb89e113640 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f25df8c1d0 0x55f25df8e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.335+0000 7fb881ffb640 10 client.15825 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.335+0000 7fb89e113640 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f25df8c1d0 0x55f25df8e5f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=988162c14af1f869 server_cookie=7488ea9c3fa3a685 in_seq=0 out_seq=0 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb881ffb640 1 -- 172.21.15.106:0/1877920865 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f25de823a0 con 0x55f25df8c1d0 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb881ffb640 10 client.15825 handle_command_reply: tid=0 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 client.15825 shutdown 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 -- 172.21.15.106:0/1877920865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df17030 msgr2=0x55f25df33010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df17030 0x55f25df33010 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb89400b930 tx=0x7fb89401b000 comp rx=0 tx=0).stop 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 -- 172.21.15.106:0/1877920865 shutdown_connections 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55f25df8c1d0 0x55f25df8e5f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25df1fe00 0x55f25df33570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f25df17030 0x55f25df33010 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb881ffb640 0 client.15825 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 --2- 172.21.15.106:0/1877920865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f25df16710 0x55f25df32ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 -- 172.21.15.106:0/1877920865 >> 172.21.15.106:0/1877920865 conn(0x55f25df11030 msgr2=0x55f25df12430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 -- 172.21.15.106:0/1877920865 shutdown_connections 2024-03-28T02:13:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 1 -- 172.21.15.106:0/1877920865 wait complete. 2024-03-28T02:13:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.359+0000 7fb8a303a740 20 client.15825 trim_cache size 0 max 16384 2024-03-28T02:13:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:13:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:13:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:13:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:13:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 -- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb8840a2a90 msgr2=0x7fb8840a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb8840a2a90 0x7fb8840a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb8980731c0 tx=0x7fb89400bb30 comp rx=0 tx=0).stop 2024-03-28T02:13:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 -- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89811ae80 msgr2=0x7fb8981ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89811ae80 0x7fb8981ab670 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb89000b9e0 tx=0x7fb89000bea0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 -- 172.21.15.106:0/380123969 shutdown_connections 2024-03-28T02:13:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb8840a2a90 0x7fb8840a4f50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb89811ae80 0x7fb8981ab670 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb898110250 0x7fb8981a72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 --2- 172.21.15.106:0/380123969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89810fde0 0x7fb8981a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 -- 172.21.15.106:0/380123969 >> 172.21.15.106:0/380123969 conn(0x7fb89806c740 msgr2=0x7fb898119870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 -- 172.21.15.106:0/380123969 shutdown_connections 2024-03-28T02:13:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:43.360+0000 7fb8a0377640 1 -- 172.21.15.106:0/380123969 wait complete. 2024-03-28T02:13:43.356 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:43.356 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 rewriters = 959480.50 kB/sec 2024-03-28T02:13:43.356 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 rewriters = 959458.24 kB/sec 2024-03-28T02:13:43.356 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 959480.50 kB/sec 2024-03-28T02:13:43.356 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 959480.50 kB/sec 2024-03-28T02:13:43.356 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 959480.50 kB/sec 2024-03-28T02:13:43.356 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 10485760.00 kB 2024-03-28T02:13:43.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:43 smithi106 ceph-mon[31965]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 725 MiB/s wr, 331 op/s 2024-03-28T02:13:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:43 smithi178 ceph-mon[36521]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 725 MiB/s wr, 331 op/s 2024-03-28T02:13:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:43 smithi153 ceph-mon[36228]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 725 MiB/s wr, 331 op/s 2024-03-28T02:13:45.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:45 smithi106 ceph-mon[31965]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 778 MiB/s wr, 339 op/s 2024-03-28T02:13:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:45 smithi178 ceph-mon[36521]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 778 MiB/s wr, 339 op/s 2024-03-28T02:13:45.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:45 smithi153 ceph-mon[36228]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 778 MiB/s wr, 339 op/s 2024-03-28T02:13:47.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:47 smithi106 ceph-mon[31965]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 778 MiB/s wr, 339 op/s 2024-03-28T02:13:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:47 smithi178 ceph-mon[36521]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 778 MiB/s wr, 339 op/s 2024-03-28T02:13:47.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:47 smithi153 ceph-mon[36228]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 778 MiB/s wr, 339 op/s 2024-03-28T02:13:49.605 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:49.605 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 readers = 7959573.50 kB/sec 2024-03-28T02:13:49.605 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 readers = 7958191.29 kB/sec 2024-03-28T02:13:49.606 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 7959573.50 kB/sec 2024-03-28T02:13:49.606 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 7959573.50 kB/sec 2024-03-28T02:13:49.606 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 7959573.50 kB/sec 2024-03-28T02:13:49.606 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 10485760.00 kB 2024-03-28T02:13:49.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:49 smithi106 ceph-mon[31965]: pgmap v79: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 853 MiB/s wr, 365 op/s 2024-03-28T02:13:49.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:49 smithi178 ceph-mon[36521]: pgmap v79: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 853 MiB/s wr, 365 op/s 2024-03-28T02:13:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:49 smithi153 ceph-mon[36228]: pgmap v79: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 853 MiB/s wr, 365 op/s 2024-03-28T02:13:51.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:51 smithi106 ceph-mon[31965]: pgmap v80: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 497 MiB/s wr, 170 op/s 2024-03-28T02:13:51.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:51 smithi178 ceph-mon[36521]: pgmap v80: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 497 MiB/s wr, 170 op/s 2024-03-28T02:13:51.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:51 smithi153 ceph-mon[36228]: pgmap v80: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 497 MiB/s wr, 170 op/s 2024-03-28T02:13:53.024 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:13:53.024 INFO:tasks.workunit.client.0.smithi106.stdout: Children see throughput for 1 re-readers = 7940485.00 kB/sec 2024-03-28T02:13:53.024 INFO:tasks.workunit.client.0.smithi106.stdout: Parent sees throughput for 1 re-readers = 7939179.57 kB/sec 2024-03-28T02:13:53.024 INFO:tasks.workunit.client.0.smithi106.stdout: Min throughput per process = 7940485.00 kB/sec 2024-03-28T02:13:53.024 INFO:tasks.workunit.client.0.smithi106.stdout: Max throughput per process = 7940485.00 kB/sec 2024-03-28T02:13:53.024 INFO:tasks.workunit.client.0.smithi106.stdout: Avg throughput per process = 7940485.00 kB/sec 2024-03-28T02:13:53.024 INFO:tasks.workunit.client.0.smithi106.stdout: Min xfer = 10485760.00 kB 2024-03-28T02:13:53.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:53 smithi106 ceph-mon[31965]: pgmap v81: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 497 MiB/s wr, 173 op/s 2024-03-28T02:13:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:53 smithi178 ceph-mon[36521]: pgmap v81: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 497 MiB/s wr, 173 op/s 2024-03-28T02:13:53.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:53 smithi153 ceph-mon[36228]: pgmap v81: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 497 MiB/s wr, 173 op/s 2024-03-28T02:13:55.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:55 smithi106 ceph-mon[31965]: pgmap v82: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 281 MiB/s wr, 82 op/s 2024-03-28T02:13:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:55 smithi178 ceph-mon[36521]: pgmap v82: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 281 MiB/s wr, 82 op/s 2024-03-28T02:13:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:55 smithi153 ceph-mon[36228]: pgmap v82: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 281 MiB/s wr, 82 op/s 2024-03-28T02:13:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.458+0000 7f9580790640 1 -- 172.21.15.106:0/3914862176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 msgr2=0x7f957810e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.458+0000 7f9580790640 1 --2- 172.21.15.106:0/3914862176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 0x7f957810e4f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9570009ca0 tx=0x7f957002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.458+0000 7f9580790640 1 -- 172.21.15.106:0/3914862176 shutdown_connections 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.458+0000 7f9580790640 1 --2- 172.21.15.106:0/3914862176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 0x7f957810e4f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.458+0000 7f9580790640 1 --2- 172.21.15.106:0/3914862176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9578068d20 0x7f9578069180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.458+0000 7f9580790640 1 --2- 172.21.15.106:0/3914862176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9578107710 0x7f9578107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.458+0000 7f9580790640 1 -- 172.21.15.106:0/3914862176 >> 172.21.15.106:0/3914862176 conn(0x7f95780773f0 msgr2=0x7f9578077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.458+0000 7f9580790640 1 -- 172.21.15.106:0/3914862176 shutdown_connections 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.458+0000 7f9580790640 1 -- 172.21.15.106:0/3914862176 wait complete. 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f9580790640 1 Processor -- start 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f9580790640 1 -- start start 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f9580790640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9578068d20 0x7f95781008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f9580790640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 0x7f9578100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f9580790640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9578107710 0x7f9578101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f9580790640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9578111a30 con 0x7f95780696c0 2024-03-28T02:13:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f9580790640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95781118b0 con 0x7f9578068d20 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f9580790640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9578111bb0 con 0x7f9578107710 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957dd2b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 0x7f9578100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957e52c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9578068d20 0x7f95781008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957dd2b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 0x7f9578100e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53416/0 (socket says 172.21.15.106:53416) 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957dd2b640 1 -- 172.21.15.106:0/3342325560 learned_addr learned my addr 172.21.15.106:0/3342325560 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957ed2d640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9578107710 0x7f9578101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957dd2b640 1 -- 172.21.15.106:0/3342325560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9578107710 msgr2=0x7f9578101340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957dd2b640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9578107710 0x7f9578101340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957dd2b640 1 -- 172.21.15.106:0/3342325560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9578068d20 msgr2=0x7f95781008c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957dd2b640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9578068d20 0x7f95781008c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957dd2b640 1 -- 172.21.15.106:0/3342325560 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95781a63b0 con 0x7f95780696c0 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.459+0000 7f957e52c640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9578068d20 0x7f95781008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.460+0000 7f957dd2b640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 0x7f9578100e00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f956800ea60 tx=0x7f956800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.460+0000 7f956f7fe640 1 -- 172.21.15.106:0/3342325560 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956800cde0 con 0x7f95780696c0 2024-03-28T02:13:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.460+0000 7f956f7fe640 1 -- 172.21.15.106:0/3342325560 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95680035f0 con 0x7f95780696c0 2024-03-28T02:13:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.460+0000 7f956f7fe640 1 -- 172.21.15.106:0/3342325560 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9568010670 con 0x7f95780696c0 2024-03-28T02:13:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.460+0000 7f9580790640 1 -- 172.21.15.106:0/3342325560 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95781a66a0 con 0x7f95780696c0 2024-03-28T02:13:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.460+0000 7f9580790640 1 -- 172.21.15.106:0/3342325560 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95781a6ba0 con 0x7f95780696c0 2024-03-28T02:13:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f956f7fe640 1 -- 172.21.15.106:0/3342325560 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f95680037a0 con 0x7f95780696c0 2024-03-28T02:13:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f9583380740 1 Processor -- start 2024-03-28T02:13:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f9583380740 1 -- start start 2024-03-28T02:13:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f9583380740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f9583380740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4c030 0x55b8e5b548c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f9583380740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b54e00 0x55b8e5b571f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f9583380740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8e5aafb10 con 0x55b8e5b4b710 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f9583380740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8e5a58900 con 0x55b8e5b54e00 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f9583380740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8e5aabbb0 con 0x55b8e5b4c030 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f957ed2d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f957ed2d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53432/0 (socket says 172.21.15.106:53432) 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f957ed2d640 1 -- 172.21.15.106:0/854704196 learned_addr learned my addr 172.21.15.106:0/854704196 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f957e52c640 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4c030 0x55b8e5b548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.463+0000 7f957dd2b640 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b54e00 0x55b8e5b571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f956f7fe640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f95640a2a90 0x7f95640a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f957e52c640 1 -- 172.21.15.106:0/854704196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b54e00 msgr2=0x55b8e5b571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f957e52c640 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b54e00 0x55b8e5b571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f957e52c640 1 -- 172.21.15.106:0/854704196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b4b710 msgr2=0x55b8e5b4baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f957e52c640 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f957e52c640 1 -- 172.21.15.106:0/854704196 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8e57c4210 con 0x55b8e5b4c030 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f957dd2b640 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b54e00 0x55b8e5b571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f957ed2d640 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f957ed2d640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f95640a2a90 0x7f95640a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f956f7fe640 1 -- 172.21.15.106:0/3342325560 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f95680cf450 con 0x7f95780696c0 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f957e52c640 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4c030 0x55b8e5b548c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9574009fb0 tx=0x7f95740118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f07c3180913ed975 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f956cff9640 1 -- 172.21.15.106:0/854704196 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9574004310 con 0x55b8e5b4c030 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f956cff9640 1 -- 172.21.15.106:0/854704196 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95740044b0 con 0x55b8e5b4c030 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.464+0000 7f956cff9640 1 -- 172.21.15.106:0/854704196 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95740035f0 con 0x55b8e5b4c030 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 -- 172.21.15.106:0/854704196 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4c030 msgr2=0x55b8e5b548c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f957ed2d640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f95640a2a90 0x7f95640a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9570009a60 tx=0x7f957002e040 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4c030 0x55b8e5b548c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9574009fb0 tx=0x7f95740118d0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 -- 172.21.15.106:0/854704196 shutdown_connections 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b54e00 0x55b8e5b571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4c030 0x55b8e5b548c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 --2- 172.21.15.106:0/854704196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 -- 172.21.15.106:0/854704196 >> 172.21.15.106:0/854704196 conn(0x55b8e5b46030 msgr2=0x55b8e5b46410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 -- 172.21.15.106:0/854704196 shutdown_connections 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 -- 172.21.15.106:0/854704196 wait complete. 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 -- start start 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.465+0000 7f9583380740 1 Processor -- start 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f9583380740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b51db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f9583380740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b522f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f9583380740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b52830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f9583380740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8e5b57a30 con 0x55b8e5b54e00 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f9583380740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8e5a6bd50 con 0x55b8e5b4c030 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f9583380740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8e5b57bb0 con 0x55b8e5b4b710 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f9583380740 10 client.0 _ll_register_callbacks cb 0x55b8e5a6b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957ed2d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b52830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957e52c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b51db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957dd2b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957ed2d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b52830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53440/0 (socket says 172.21.15.106:53440) 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957ed2d640 1 -- 172.21.15.106:0/1383884651 learned_addr learned my addr 172.21.15.106:0/1383884651 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957e52c640 1 -- 172.21.15.106:0/1383884651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 msgr2=0x55b8e5b522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957e52c640 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b522f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957e52c640 1 -- 172.21.15.106:0/1383884651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 msgr2=0x55b8e5b52830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957e52c640 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b52830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957e52c640 1 -- 172.21.15.106:0/1383884651 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8e5890620 con 0x55b8e5b4b710 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957ed2d640 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b52830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f957e52c640 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b51db0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f957401b040 tx=0x7f9574004970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed169b2a99f35f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.466+0000 7f9561ffb640 10 client.0 ms_handle_connect on v2:172.21.15.178:3300/0 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9561ffb640 1 -- 172.21.15.106:0/1383884651 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9574029070 con 0x55b8e5b4b710 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9583380740 1 -- 172.21.15.106:0/1383884651 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95740044b0 con 0x55b8e5b4b710 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9561ffb640 1 -- 172.21.15.106:0/1383884651 <== mon.1 v2:172.21.15.178:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f9574003ac0 con 0x55b8e5b4b710 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9561ffb640 1 -- 172.21.15.106:0/1383884651 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9574035050 con 0x55b8e5b4b710 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9583380740 10 client.25597 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9583380740 1 -- 172.21.15.106:0/1383884651 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b8e5b63c10 con 0x55b8e5b4b710 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9561ffb640 1 -- 172.21.15.106:0/1383884651 <== mon.1 v2:172.21.15.178:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95740044b0 con 0x55b8e5b4b710 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9561ffb640 1 -- 172.21.15.106:0/1383884651 <== mon.1 v2:172.21.15.178:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f957400f480 con 0x55b8e5b4b710 2024-03-28T02:13:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9583380740 10 client.25597 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9583380740 10 client.25597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9583380740 20 client.25597 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9583380740 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55b8e5b65020 0x55b8e5b67410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9583380740 4 client.25597 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f9583380740 1 -- 172.21.15.106:0/1383884651 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8e5abd050 con 0x55b8e5b65020 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.467+0000 7f957dd2b640 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55b8e5b65020 0x55b8e5b67410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.468+0000 7f957dd2b640 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55b8e5b65020 0x55b8e5b67410 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=544c515a6bb230ba server_cookie=b5689dd679052ac6 in_seq=0 out_seq=0 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.468+0000 7f9561ffb640 10 client.25597 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9561ffb640 1 -- 172.21.15.106:0/1383884651 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b8e5abd050 con 0x55b8e5b65020 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9561ffb640 10 client.25597 handle_command_reply: tid=0 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 client.25597 shutdown 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 -- 172.21.15.106:0/1383884651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 msgr2=0x55b8e5b51db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b51db0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f957401b040 tx=0x7f9574004970 comp rx=0 tx=0).stop 2024-03-28T02:13:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 -- 172.21.15.106:0/1383884651 shutdown_connections 2024-03-28T02:13:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55b8e5b65020 0x55b8e5b67410 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b52830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 --2- 172.21.15.106:0/1383884651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b51db0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9561ffb640 0 client.25597 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 -- 172.21.15.106:0/1383884651 >> 172.21.15.106:0/1383884651 conn(0x55b8e5b46030 msgr2=0x55b8e5b47d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 -- 172.21.15.106:0/1383884651 shutdown_connections 2024-03-28T02:13:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 1 -- 172.21.15.106:0/1383884651 wait complete. 2024-03-28T02:13:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.469+0000 7f9583380740 20 client.25597 trim_cache size 0 max 16384 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9583380740 1 Processor -- start 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9583380740 1 -- start start 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9583380740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b544e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9583380740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b50d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9583380740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b602c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9583380740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95781b0a50 con 0x55b8e5b54e00 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9583380740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95781b23d0 con 0x55b8e5b4c030 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9583380740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95781b10b0 con 0x55b8e5b4b710 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957ed2d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957ed2d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b602c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53446/0 (socket says 172.21.15.106:53446) 2024-03-28T02:13:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957ed2d640 1 -- 172.21.15.106:0/2337332953 learned_addr learned my addr 172.21.15.106:0/2337332953 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957e52c640 1 --2- 172.21.15.106:0/2337332953 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957dd2b640 1 --2- 172.21.15.106:0/2337332953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b50d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957ed2d640 1 -- 172.21.15.106:0/2337332953 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 msgr2=0x55b8e5b544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957ed2d640 1 --2- 172.21.15.106:0/2337332953 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957ed2d640 1 -- 172.21.15.106:0/2337332953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 msgr2=0x55b8e5b50d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957ed2d640 1 --2- 172.21.15.106:0/2337332953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b50d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957ed2d640 1 -- 172.21.15.106:0/2337332953 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8e5b4be60 con 0x55b8e5b54e00 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f957ed2d640 1 --2- 172.21.15.106:0/2337332953 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b602c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f95700096a0 tx=0x7f9570041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e09a6005ebe072c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9563fff640 1 -- 172.21.15.106:0/2337332953 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570039040 con 0x55b8e5b54e00 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9563fff640 1 -- 172.21.15.106:0/2337332953 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9570052070 con 0x55b8e5b54e00 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.483+0000 7f9563fff640 1 -- 172.21.15.106:0/2337332953 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957004daa0 con 0x55b8e5b54e00 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 -- 172.21.15.106:0/2337332953 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 msgr2=0x55b8e5b602c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 --2- 172.21.15.106:0/2337332953 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b602c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f95700096a0 tx=0x7f9570041000 comp rx=0 tx=0).stop 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 -- 172.21.15.106:0/2337332953 shutdown_connections 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 --2- 172.21.15.106:0/2337332953 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b602c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 --2- 172.21.15.106:0/2337332953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b50d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 --2- 172.21.15.106:0/2337332953 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 -- 172.21.15.106:0/2337332953 >> 172.21.15.106:0/2337332953 conn(0x55b8e5b46030 msgr2=0x55b8e5b50680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 -- 172.21.15.106:0/2337332953 shutdown_connections 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 -- 172.21.15.106:0/2337332953 wait complete. 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 -- start start 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.484+0000 7f9583380740 1 Processor -- start 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b4e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b4e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95781b1d70 con 0x55b8e5b54e00 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95781b1710 con 0x55b8e5b4c030 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95781aec10 con 0x55b8e5b4b710 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 10 client.0 _ll_register_callbacks cb 0x7f9574009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957ed2d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b4e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957e52c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957dd2b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b4e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957ed2d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b4e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:53462/0 (socket says 172.21.15.106:53462) 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957ed2d640 1 -- 172.21.15.106:0/3687175559 learned_addr learned my addr 172.21.15.106:0/3687175559 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957e52c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:37316/0 (socket says 172.21.15.106:37316) 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957dd2b640 1 -- 172.21.15.106:0/3687175559 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 msgr2=0x55b8e5b4f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957dd2b640 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957dd2b640 1 -- 172.21.15.106:0/3687175559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 msgr2=0x55b8e5b4e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957dd2b640 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b4e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957dd2b640 1 -- 172.21.15.106:0/3687175559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8e5b4eaa0 con 0x55b8e5b4c030 2024-03-28T02:13:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957ed2d640 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b4e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f957dd2b640 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b4e020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f95680cc820 tx=0x7f95680d5060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=591bfe9e95641751 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9561ffb640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9561ffb640 1 -- 172.21.15.106:0/3687175559 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956809b090 con 0x55b8e5b4c030 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 1 -- 172.21.15.106:0/3687175559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9570052070 con 0x55b8e5b4c030 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9561ffb640 1 -- 172.21.15.106:0/3687175559 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f9568003b30 con 0x55b8e5b4c030 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9561ffb640 1 -- 172.21.15.106:0/3687175559 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f956809be30 con 0x55b8e5b4c030 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 10 client.25697 fetch_fsmap learned FSMap version 29 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.485+0000 7f9583380740 1 -- 172.21.15.106:0/3687175559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b8e5bc03e0 con 0x55b8e5b4c030 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f9561ffb640 1 -- 172.21.15.106:0/3687175559 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9570052070 con 0x55b8e5b4c030 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f9561ffb640 1 -- 172.21.15.106:0/3687175559 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f95680ccaa0 con 0x55b8e5b4c030 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f9583380740 10 client.25697 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f9583380740 10 client.25697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f9583380740 20 client.25697 populate_metadata read hostname 'smithi106' 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f9583380740 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55b8e5bc0b30 0x55b8e5bc2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:13:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f9583380740 4 client.25697 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T02:13:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f9583380740 1 -- 172.21.15.106:0/3687175559 --> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b8e5ab73a0 con 0x55b8e5bc0b30 2024-03-28T02:13:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f957e52c640 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55b8e5bc0b30 0x55b8e5bc2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:13:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f957e52c640 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55b8e5bc0b30 0x55b8e5bc2f50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83060e45f910442d server_cookie=e000e31e347db894 in_seq=0 out_seq=0 2024-03-28T02:13:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.486+0000 7f9561ffb640 10 client.25697 ms_handle_connect on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9561ffb640 1 -- 172.21.15.106:0/3687175559 <== mds.0 v2:172.21.15.153:6838/505906484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b8e5ab73a0 con 0x55b8e5bc0b30 2024-03-28T02:13:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9561ffb640 10 client.25697 handle_command_reply: tid=0 2024-03-28T02:13:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 client.25697 shutdown 2024-03-28T02:13:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 -- 172.21.15.106:0/3687175559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 msgr2=0x55b8e5b4e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b4e020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f95680cc820 tx=0x7f95680d5060 comp rx=0 tx=0).stop 2024-03-28T02:13:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 -- 172.21.15.106:0/3687175559 shutdown_connections 2024-03-28T02:13:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.153:6838/505906484,v1:172.21.15.153:6839/505906484] conn(0x55b8e5bc0b30 0x55b8e5bc2f50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8e5b54e00 0x55b8e5b4e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8e5b4c030 0x55b8e5b4e020 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9561ffb640 0 client.25697 ms_handle_reset on v2:172.21.15.153:6838/505906484 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 --2- 172.21.15.106:0/3687175559 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b8e5b4b710 0x55b8e5b4f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 -- 172.21.15.106:0/3687175559 >> 172.21.15.106:0/3687175559 conn(0x55b8e5b46030 msgr2=0x55b8e5b50f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 -- 172.21.15.106:0/3687175559 shutdown_connections 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 1 -- 172.21.15.106:0/3687175559 wait complete. 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9583380740 20 client.25697 trim_cache size 0 max 16384 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:{ 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "message": "", 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout: "return_code": 0 2024-03-28T02:13:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stdout:} 2024-03-28T02:13:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9580790640 1 -- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f95640a2a90 msgr2=0x7f95640a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9580790640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f95640a2a90 0x7f95640a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9570009a60 tx=0x7f957002e040 comp rx=0 tx=0).stop 2024-03-28T02:13:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9580790640 1 -- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 msgr2=0x7f9578100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:13:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.495+0000 7f9580790640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 0x7f9578100e00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f956800ea60 tx=0x7f956800ef20 comp rx=0 tx=0).stop 2024-03-28T02:13:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.496+0000 7f9580790640 1 -- 172.21.15.106:0/3342325560 shutdown_connections 2024-03-28T02:13:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.496+0000 7f9580790640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f95640a2a90 0x7f95640a4f50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.496+0000 7f9580790640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9578107710 0x7f9578101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.496+0000 7f9580790640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780696c0 0x7f9578100e00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.496+0000 7f9580790640 1 --2- 172.21.15.106:0/3342325560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9578068d20 0x7f95781008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:13:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.496+0000 7f9580790640 1 -- 172.21.15.106:0/3342325560 >> 172.21.15.106:0/3342325560 conn(0x7f95780773f0 msgr2=0x7f95780ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:13:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.496+0000 7f9580790640 1 -- 172.21.15.106:0/3342325560 shutdown_connections 2024-03-28T02:13:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi106.front.sepia.ceph.com.smithi106.stderr:2024-03-28T02:13:56.496+0000 7f9580790640 1 -- 172.21.15.106:0/3342325560 wait complete. 2024-03-28T02:13:57.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:57 smithi106 ceph-mon[31965]: pgmap v83: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 75 MiB/s wr, 29 op/s 2024-03-28T02:13:57.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:57 smithi178 ceph-mon[36521]: pgmap v83: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 75 MiB/s wr, 29 op/s 2024-03-28T02:13:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:57 smithi153 ceph-mon[36228]: pgmap v83: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 75 MiB/s wr, 29 op/s 2024-03-28T02:13:59.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:13:59 smithi106 ceph-mon[31965]: pgmap v84: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 75 MiB/s wr, 36 op/s 2024-03-28T02:13:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:13:59 smithi178 ceph-mon[36521]: pgmap v84: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 75 MiB/s wr, 36 op/s 2024-03-28T02:13:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:13:59 smithi153 ceph-mon[36228]: pgmap v84: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 75 MiB/s wr, 36 op/s 2024-03-28T02:14:00.320 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:14:00.320 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:14:00.320 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-03-28T02:14:00.320 INFO:tasks.workunit.client.0.smithi106.stdout:iozone test complete. 2024-03-28T02:14:00.322 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T02:14:00.322 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T02:14:00.410 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-03-28T02:14:00.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T02:14:00.856 DEBUG:teuthology.parallel:result is None 2024-03-28T02:14:00.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T02:14:00.905 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T02:14:00.906 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-28T02:14:00.917 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:14:00.929 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-28T02:14:00.938 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-28T02:14:00.938 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T02:14:00.938 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:14:00.938 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-28T02:14:01.008 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-28T02:14:01.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:01 smithi106 ceph-mon[31965]: pgmap v85: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.8 KiB/s wr, 11 op/s 2024-03-28T02:14:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:01 smithi178 ceph-mon[36521]: pgmap v85: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.8 KiB/s wr, 11 op/s 2024-03-28T02:14:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:01 smithi153 ceph-mon[36228]: pgmap v85: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.8 KiB/s wr, 11 op/s 2024-03-28T02:14:03.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:03 smithi106 ceph-mon[31965]: pgmap v86: 129 pgs: 129 active+clean; 4.8 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 124 op/s 2024-03-28T02:14:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:03 smithi178 ceph-mon[36521]: pgmap v86: 129 pgs: 129 active+clean; 4.8 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 124 op/s 2024-03-28T02:14:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:03 smithi153 ceph-mon[36228]: pgmap v86: 129 pgs: 129 active+clean; 4.8 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 124 op/s 2024-03-28T02:14:05.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:05 smithi106 ceph-mon[31965]: pgmap v87: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 KiB/s wr, 241 op/s 2024-03-28T02:14:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:05 smithi178 ceph-mon[36521]: pgmap v87: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 KiB/s wr, 241 op/s 2024-03-28T02:14:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:05 smithi153 ceph-mon[36228]: pgmap v87: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 KiB/s wr, 241 op/s 2024-03-28T02:14:06.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:06 smithi178 ceph-mon[36521]: pgmap v88: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 28 KiB/s wr, 240 op/s 2024-03-28T02:14:06.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:06 smithi153 ceph-mon[36228]: pgmap v88: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 28 KiB/s wr, 240 op/s 2024-03-28T02:14:07.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:06 smithi106 ceph-mon[31965]: pgmap v88: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 28 KiB/s wr, 240 op/s 2024-03-28T02:14:09.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:09 smithi106 ceph-mon[31965]: pgmap v89: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 29 KiB/s wr, 244 op/s 2024-03-28T02:14:09.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:09 smithi178 ceph-mon[36521]: pgmap v89: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 29 KiB/s wr, 244 op/s 2024-03-28T02:14:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:09 smithi153 ceph-mon[36228]: pgmap v89: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 29 KiB/s wr, 244 op/s 2024-03-28T02:14:11.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:11 smithi106 ceph-mon[31965]: pgmap v90: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 238 op/s 2024-03-28T02:14:11.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:11 smithi178 ceph-mon[36521]: pgmap v90: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 238 op/s 2024-03-28T02:14:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:11 smithi153 ceph-mon[36228]: pgmap v90: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 238 op/s 2024-03-28T02:14:13.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:13 smithi106 ceph-mon[31965]: pgmap v91: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 KiB/s wr, 241 op/s 2024-03-28T02:14:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:13 smithi178 ceph-mon[36521]: pgmap v91: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 KiB/s wr, 241 op/s 2024-03-28T02:14:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:13 smithi153 ceph-mon[36228]: pgmap v91: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 KiB/s wr, 241 op/s 2024-03-28T02:14:15.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:15 smithi106 ceph-mon[31965]: pgmap v92: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 128 op/s 2024-03-28T02:14:15.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:15 smithi178 ceph-mon[36521]: pgmap v92: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 128 op/s 2024-03-28T02:14:15.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:15 smithi153 ceph-mon[36228]: pgmap v92: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 128 op/s 2024-03-28T02:14:17.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:17 smithi106 ceph-mon[31965]: pgmap v93: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T02:14:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:17 smithi178 ceph-mon[36521]: pgmap v93: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T02:14:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:17 smithi153 ceph-mon[36228]: pgmap v93: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T02:14:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:19 smithi106 ceph-mon[31965]: pgmap v94: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-28T02:14:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:19 smithi178 ceph-mon[36521]: pgmap v94: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-28T02:14:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:19 smithi153 ceph-mon[36228]: pgmap v94: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-28T02:14:21.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:21 smithi106 ceph-mon[31965]: pgmap v95: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-28T02:14:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:21 smithi178 ceph-mon[36521]: pgmap v95: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-28T02:14:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:21 smithi153 ceph-mon[36228]: pgmap v95: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-28T02:14:23.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:23 smithi106 ceph-mon[31965]: pgmap v96: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1023 B/s wr, 11 op/s 2024-03-28T02:14:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:23 smithi178 ceph-mon[36521]: pgmap v96: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1023 B/s wr, 11 op/s 2024-03-28T02:14:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:23 smithi153 ceph-mon[36228]: pgmap v96: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1023 B/s wr, 11 op/s 2024-03-28T02:14:25.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:25 smithi106 ceph-mon[31965]: pgmap v97: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:25.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:25 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:14:25.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:25 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:14:25.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:25 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:14:25.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:25 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:14:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:25 smithi178 ceph-mon[36521]: pgmap v97: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:25 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:14:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:25 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:14:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:25 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:14:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:25 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:14:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:25 smithi153 ceph-mon[36228]: pgmap v97: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:25 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:14:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:25 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:14:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:25 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:14:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:25 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:14:27.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:27 smithi153 ceph-mon[36228]: pgmap v98: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:27.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:27 smithi106 ceph-mon[31965]: pgmap v98: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:27 smithi178 ceph-mon[36521]: pgmap v98: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:29.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:29 smithi106 ceph-mon[31965]: pgmap v99: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:29 smithi178 ceph-mon[36521]: pgmap v99: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:29.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:29 smithi153 ceph-mon[36228]: pgmap v99: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:31.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:31 smithi106 ceph-mon[31965]: pgmap v100: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:31 smithi178 ceph-mon[36521]: pgmap v100: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:31.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:31 smithi153 ceph-mon[36228]: pgmap v100: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:33.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:33 smithi106 ceph-mon[31965]: pgmap v101: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T02:14:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:33 smithi178 ceph-mon[36521]: pgmap v101: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T02:14:33.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:33 smithi153 ceph-mon[36228]: pgmap v101: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T02:14:35.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:35 smithi106 ceph-mon[31965]: pgmap v102: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:35 smithi178 ceph-mon[36521]: pgmap v102: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:35 smithi153 ceph-mon[36228]: pgmap v102: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:37.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:37 smithi106 ceph-mon[31965]: pgmap v103: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:37 smithi178 ceph-mon[36521]: pgmap v103: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:37.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:37 smithi153 ceph-mon[36228]: pgmap v103: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:39.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:39 smithi106 ceph-mon[31965]: pgmap v104: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:39 smithi178 ceph-mon[36521]: pgmap v104: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:39.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:39 smithi153 ceph-mon[36228]: pgmap v104: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:41.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:41 smithi106 ceph-mon[31965]: pgmap v105: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:41 smithi178 ceph-mon[36521]: pgmap v105: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:41.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:41 smithi153 ceph-mon[36228]: pgmap v105: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:43.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:43 smithi106 ceph-mon[31965]: pgmap v106: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T02:14:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:43 smithi178 ceph-mon[36521]: pgmap v106: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T02:14:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:43 smithi153 ceph-mon[36228]: pgmap v106: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T02:14:45.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:45 smithi106 ceph-mon[31965]: pgmap v107: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:45 smithi178 ceph-mon[36521]: pgmap v107: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:45.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:45 smithi153 ceph-mon[36228]: pgmap v107: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:47.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:47 smithi106 ceph-mon[31965]: pgmap v108: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:47 smithi178 ceph-mon[36521]: pgmap v108: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:47.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:47 smithi153 ceph-mon[36228]: pgmap v108: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:49.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:49 smithi106 ceph-mon[31965]: pgmap v109: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:49.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:49 smithi178 ceph-mon[36521]: pgmap v109: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:49 smithi153 ceph-mon[36228]: pgmap v109: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:51.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:51 smithi106 ceph-mon[31965]: pgmap v110: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:51 smithi178 ceph-mon[36521]: pgmap v110: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:51.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:51 smithi153 ceph-mon[36228]: pgmap v110: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:53.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:53 smithi106 ceph-mon[31965]: pgmap v111: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T02:14:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:53 smithi178 ceph-mon[36521]: pgmap v111: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T02:14:53.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:53 smithi153 ceph-mon[36228]: pgmap v111: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T02:14:55.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:55 smithi106 ceph-mon[31965]: pgmap v112: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:55 smithi178 ceph-mon[36521]: pgmap v112: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:55 smithi153 ceph-mon[36228]: pgmap v112: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T02:14:57.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:57 smithi106 ceph-mon[31965]: pgmap v113: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:57.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:57 smithi178 ceph-mon[36521]: pgmap v113: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:57 smithi153 ceph-mon[36228]: pgmap v113: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:14:59.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:14:59 smithi106 ceph-mon[31965]: pgmap v114: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:14:59 smithi178 ceph-mon[36521]: pgmap v114: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:14:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:14:59 smithi153 ceph-mon[36228]: pgmap v114: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:01.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:01 smithi106 ceph-mon[31965]: pgmap v115: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:01 smithi178 ceph-mon[36521]: pgmap v115: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:01 smithi153 ceph-mon[36228]: pgmap v115: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:03.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:03 smithi106 ceph-mon[31965]: pgmap v116: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:15:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:03 smithi178 ceph-mon[36521]: pgmap v116: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:15:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:03 smithi153 ceph-mon[36228]: pgmap v116: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T02:15:05.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:05 smithi106 ceph-mon[31965]: pgmap v117: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T02:15:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:05 smithi178 ceph-mon[36521]: pgmap v117: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T02:15:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:05 smithi153 ceph-mon[36228]: pgmap v117: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T02:15:07.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:07 smithi106 ceph-mon[31965]: pgmap v118: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:15:07.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:07 smithi178 ceph-mon[36521]: pgmap v118: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:15:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:07 smithi153 ceph-mon[36228]: pgmap v118: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:15:08.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:08 smithi178 ceph-mon[36521]: pgmap v119: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:15:08.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:08 smithi153 ceph-mon[36228]: pgmap v119: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:15:09.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:08 smithi106 ceph-mon[31965]: pgmap v119: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:15:11.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:11 smithi106 ceph-mon[31965]: pgmap v120: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:15:11.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:11 smithi178 ceph-mon[36521]: pgmap v120: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:15:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:11 smithi153 ceph-mon[36228]: pgmap v120: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:15:13.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:13 smithi106 ceph-mon[31965]: pgmap v121: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T02:15:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:13 smithi178 ceph-mon[36521]: pgmap v121: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T02:15:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:13 smithi153 ceph-mon[36228]: pgmap v121: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T02:15:15.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:15 smithi106 ceph-mon[31965]: pgmap v122: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T02:15:15.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:15 smithi178 ceph-mon[36521]: pgmap v122: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T02:15:15.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:15 smithi153 ceph-mon[36228]: pgmap v122: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T02:15:17.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:17 smithi106 ceph-mon[31965]: pgmap v123: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T02:15:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:17 smithi178 ceph-mon[36521]: pgmap v123: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T02:15:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:17 smithi153 ceph-mon[36228]: pgmap v123: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T02:15:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:19 smithi106 ceph-mon[31965]: pgmap v124: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T02:15:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:19 smithi178 ceph-mon[36521]: pgmap v124: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T02:15:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:19 smithi153 ceph-mon[36228]: pgmap v124: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T02:15:21.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:21 smithi106 ceph-mon[31965]: pgmap v125: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T02:15:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:21 smithi178 ceph-mon[36521]: pgmap v125: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T02:15:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:21 smithi153 ceph-mon[36228]: pgmap v125: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T02:15:23.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:23 smithi106 ceph-mon[31965]: pgmap v126: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T02:15:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:23 smithi178 ceph-mon[36521]: pgmap v126: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T02:15:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:23 smithi153 ceph-mon[36228]: pgmap v126: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T02:15:25.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:25 smithi106 ceph-mon[31965]: pgmap v127: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T02:15:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:25 smithi178 ceph-mon[36521]: pgmap v127: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T02:15:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:25 smithi153 ceph-mon[36228]: pgmap v127: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T02:15:26.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:26 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:15:26.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:26 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:15:26.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:26 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:15:26.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:26 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:15:26.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:26 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:15:26.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:26 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:15:26.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:26 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:15:26.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:26 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:15:26.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:26 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:15:26.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:26 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:15:26.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:26 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:15:26.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:26 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:15:27.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:27 smithi106 ceph-mon[31965]: pgmap v128: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:27 smithi178 ceph-mon[36521]: pgmap v128: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:27.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:27 smithi153 ceph-mon[36228]: pgmap v128: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:29.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:29 smithi106 ceph-mon[31965]: pgmap v129: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:29 smithi178 ceph-mon[36521]: pgmap v129: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:29.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:29 smithi153 ceph-mon[36228]: pgmap v129: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:31.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:31 smithi106 ceph-mon[31965]: pgmap v130: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:31 smithi178 ceph-mon[36521]: pgmap v130: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:31.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:31 smithi153 ceph-mon[36228]: pgmap v130: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:33.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:33 smithi106 ceph-mon[31965]: pgmap v131: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:15:33.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:33 smithi178 ceph-mon[36521]: pgmap v131: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:15:33.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:33 smithi153 ceph-mon[36228]: pgmap v131: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:15:35.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:35 smithi106 ceph-mon[31965]: pgmap v132: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:15:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:35 smithi178 ceph-mon[36521]: pgmap v132: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:15:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:35 smithi153 ceph-mon[36228]: pgmap v132: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:15:37.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:37 smithi106 ceph-mon[31965]: pgmap v133: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:37 smithi178 ceph-mon[36521]: pgmap v133: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:37.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:37 smithi153 ceph-mon[36228]: pgmap v133: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:39.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:39 smithi106 ceph-mon[31965]: pgmap v134: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:39 smithi178 ceph-mon[36521]: pgmap v134: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:39.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:39 smithi153 ceph-mon[36228]: pgmap v134: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:41.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:41 smithi106 ceph-mon[31965]: pgmap v135: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:41 smithi178 ceph-mon[36521]: pgmap v135: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:41.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:41 smithi153 ceph-mon[36228]: pgmap v135: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:43.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:43 smithi106 ceph-mon[31965]: pgmap v136: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:15:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:43 smithi178 ceph-mon[36521]: pgmap v136: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:15:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:43 smithi153 ceph-mon[36228]: pgmap v136: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:15:45.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:45 smithi106 ceph-mon[31965]: pgmap v137: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:15:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:45 smithi178 ceph-mon[36521]: pgmap v137: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:15:45.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:45 smithi153 ceph-mon[36228]: pgmap v137: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:15:47.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:47 smithi106 ceph-mon[31965]: pgmap v138: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:47 smithi178 ceph-mon[36521]: pgmap v138: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:47.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:47 smithi153 ceph-mon[36228]: pgmap v138: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:49.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:49 smithi106 ceph-mon[31965]: pgmap v139: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:49.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:49 smithi178 ceph-mon[36521]: pgmap v139: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:49 smithi153 ceph-mon[36228]: pgmap v139: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:51.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:51 smithi106 ceph-mon[31965]: pgmap v140: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:51 smithi178 ceph-mon[36521]: pgmap v140: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:51.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:51 smithi153 ceph-mon[36228]: pgmap v140: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:53.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:53 smithi106 ceph-mon[31965]: pgmap v141: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:15:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:53 smithi178 ceph-mon[36521]: pgmap v141: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:15:53.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:53 smithi153 ceph-mon[36228]: pgmap v141: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:15:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:55 smithi178 ceph-mon[36521]: pgmap v142: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:15:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:55 smithi153 ceph-mon[36228]: pgmap v142: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:15:56.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:55 smithi106 ceph-mon[31965]: pgmap v142: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:15:56.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:56 smithi153 ceph-mon[36228]: pgmap v143: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:57.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:56 smithi106 ceph-mon[31965]: pgmap v143: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:57.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:56 smithi178 ceph-mon[36521]: pgmap v143: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:15:59.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:15:59 smithi106 ceph-mon[31965]: pgmap v144: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:15:59 smithi178 ceph-mon[36521]: pgmap v144: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:15:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:15:59 smithi153 ceph-mon[36228]: pgmap v144: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:01.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:01 smithi106 ceph-mon[31965]: pgmap v145: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:01 smithi178 ceph-mon[36521]: pgmap v145: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:01 smithi153 ceph-mon[36228]: pgmap v145: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:03.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:03 smithi106 ceph-mon[31965]: pgmap v146: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-28T02:16:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:03 smithi178 ceph-mon[36521]: pgmap v146: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-28T02:16:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:03 smithi153 ceph-mon[36228]: pgmap v146: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-28T02:16:05.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:05 smithi106 ceph-mon[31965]: pgmap v147: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:16:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:05 smithi178 ceph-mon[36521]: pgmap v147: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:16:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:05 smithi153 ceph-mon[36228]: pgmap v147: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:16:07.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:07 smithi106 ceph-mon[31965]: pgmap v148: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:16:07.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:07 smithi178 ceph-mon[36521]: pgmap v148: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:16:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:07 smithi153 ceph-mon[36228]: pgmap v148: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:16:09.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:09 smithi106 ceph-mon[31965]: pgmap v149: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:16:09.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:09 smithi178 ceph-mon[36521]: pgmap v149: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:16:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:09 smithi153 ceph-mon[36228]: pgmap v149: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:16:11.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:11 smithi106 ceph-mon[31965]: pgmap v150: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:16:11.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:11 smithi178 ceph-mon[36521]: pgmap v150: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:16:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:11 smithi153 ceph-mon[36228]: pgmap v150: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:16:13.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:13 smithi106 ceph-mon[31965]: pgmap v151: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:16:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:13 smithi178 ceph-mon[36521]: pgmap v151: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:16:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:13 smithi153 ceph-mon[36228]: pgmap v151: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:16:15.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:15 smithi106 ceph-mon[31965]: pgmap v152: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 22 op/s 2024-03-28T02:16:15.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:15 smithi178 ceph-mon[36521]: pgmap v152: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 22 op/s 2024-03-28T02:16:15.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:15 smithi153 ceph-mon[36228]: pgmap v152: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 22 op/s 2024-03-28T02:16:17.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:17 smithi106 ceph-mon[31965]: pgmap v153: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T02:16:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:17 smithi178 ceph-mon[36521]: pgmap v153: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T02:16:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:17 smithi153 ceph-mon[36228]: pgmap v153: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T02:16:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:19 smithi106 ceph-mon[31965]: pgmap v154: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T02:16:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:19 smithi178 ceph-mon[36521]: pgmap v154: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T02:16:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:19 smithi153 ceph-mon[36228]: pgmap v154: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T02:16:21.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:21 smithi106 ceph-mon[31965]: pgmap v155: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T02:16:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:21 smithi178 ceph-mon[36521]: pgmap v155: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T02:16:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:21 smithi153 ceph-mon[36228]: pgmap v155: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T02:16:23.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:23 smithi106 ceph-mon[31965]: pgmap v156: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T02:16:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:23 smithi178 ceph-mon[36521]: pgmap v156: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T02:16:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:23 smithi153 ceph-mon[36228]: pgmap v156: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T02:16:25.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:25 smithi106 ceph-mon[31965]: pgmap v157: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:25 smithi178 ceph-mon[36521]: pgmap v157: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:25 smithi153 ceph-mon[36228]: pgmap v157: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:26.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:26 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:16:26.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:26 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:16:26.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:26 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:16:27.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:27 smithi106 ceph-mon[31965]: pgmap v158: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:27.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:27 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:16:27.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:27 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:16:27.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:27 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:16:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:27 smithi178 ceph-mon[36521]: pgmap v158: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:27 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:16:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:27 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:16:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:27 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:16:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:27 smithi153 ceph-mon[36228]: pgmap v158: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:27 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:16:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:27 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:16:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:27 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:16:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:29 smithi178 ceph-mon[36521]: pgmap v159: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:29.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:29 smithi153 ceph-mon[36228]: pgmap v159: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:30.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:29 smithi106 ceph-mon[31965]: pgmap v159: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:31 smithi178 ceph-mon[36521]: pgmap v160: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:31.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:31 smithi153 ceph-mon[36228]: pgmap v160: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:32.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:31 smithi106 ceph-mon[31965]: pgmap v160: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:33 smithi178 ceph-mon[36521]: pgmap v161: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:16:33.884 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:33 smithi106 ceph-mon[31965]: pgmap v161: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:16:33.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:33 smithi153 ceph-mon[36228]: pgmap v161: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:16:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:35 smithi178 ceph-mon[36521]: pgmap v162: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:35 smithi153 ceph-mon[36228]: pgmap v162: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:36.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:35 smithi106 ceph-mon[31965]: pgmap v162: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:36.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:36 smithi153 ceph-mon[36228]: pgmap v163: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:37.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:36 smithi106 ceph-mon[31965]: pgmap v163: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:37.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:36 smithi178 ceph-mon[36521]: pgmap v163: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:39.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:39 smithi106 ceph-mon[31965]: pgmap v164: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:39 smithi178 ceph-mon[36521]: pgmap v164: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:39.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:39 smithi153 ceph-mon[36228]: pgmap v164: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:41.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:41 smithi106 ceph-mon[31965]: pgmap v165: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:41 smithi178 ceph-mon[36521]: pgmap v165: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:41.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:41 smithi153 ceph-mon[36228]: pgmap v165: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:43.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:43 smithi106 ceph-mon[31965]: pgmap v166: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:16:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:43 smithi178 ceph-mon[36521]: pgmap v166: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:16:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:43 smithi153 ceph-mon[36228]: pgmap v166: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:16:45.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:45 smithi106 ceph-mon[31965]: pgmap v167: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:45 smithi178 ceph-mon[36521]: pgmap v167: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:45.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:45 smithi153 ceph-mon[36228]: pgmap v167: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:47.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:47 smithi106 ceph-mon[31965]: pgmap v168: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:47 smithi178 ceph-mon[36521]: pgmap v168: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:47.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:47 smithi153 ceph-mon[36228]: pgmap v168: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:49.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:49 smithi106 ceph-mon[31965]: pgmap v169: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:49.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:49 smithi178 ceph-mon[36521]: pgmap v169: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:49 smithi153 ceph-mon[36228]: pgmap v169: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:51.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:51 smithi106 ceph-mon[31965]: pgmap v170: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:51 smithi178 ceph-mon[36521]: pgmap v170: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:51.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:51 smithi153 ceph-mon[36228]: pgmap v170: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:53.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:53 smithi106 ceph-mon[31965]: pgmap v171: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:16:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:53 smithi178 ceph-mon[36521]: pgmap v171: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:16:53.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:53 smithi153 ceph-mon[36228]: pgmap v171: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T02:16:55.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:55 smithi106 ceph-mon[31965]: pgmap v172: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:55 smithi178 ceph-mon[36521]: pgmap v172: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:55 smithi153 ceph-mon[36228]: pgmap v172: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:16:57.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:57 smithi106 ceph-mon[31965]: pgmap v173: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:57.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:57 smithi178 ceph-mon[36521]: pgmap v173: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:57 smithi153 ceph-mon[36228]: pgmap v173: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:16:59.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:16:59 smithi106 ceph-mon[31965]: pgmap v174: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:16:59 smithi178 ceph-mon[36521]: pgmap v174: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:16:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:16:59 smithi153 ceph-mon[36228]: pgmap v174: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:01.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:01 smithi106 ceph-mon[31965]: pgmap v175: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:01 smithi178 ceph-mon[36521]: pgmap v175: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:01 smithi153 ceph-mon[36228]: pgmap v175: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:03.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:03 smithi106 ceph-mon[31965]: pgmap v176: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-28T02:17:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:03 smithi178 ceph-mon[36521]: pgmap v176: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-28T02:17:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:03 smithi153 ceph-mon[36228]: pgmap v176: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-28T02:17:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:05 smithi178 ceph-mon[36521]: pgmap v177: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:17:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:05 smithi153 ceph-mon[36228]: pgmap v177: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:17:06.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:05 smithi106 ceph-mon[31965]: pgmap v177: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:17:07.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:07 smithi178 ceph-mon[36521]: pgmap v178: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:17:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:07 smithi153 ceph-mon[36228]: pgmap v178: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:17:08.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:07 smithi106 ceph-mon[31965]: pgmap v178: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:17:09.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:09 smithi178 ceph-mon[36521]: pgmap v179: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:17:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:09 smithi153 ceph-mon[36228]: pgmap v179: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:17:10.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:09 smithi106 ceph-mon[31965]: pgmap v179: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:17:11.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:11 smithi178 ceph-mon[36521]: pgmap v180: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:17:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:11 smithi153 ceph-mon[36228]: pgmap v180: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:17:12.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:11 smithi106 ceph-mon[31965]: pgmap v180: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T02:17:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:13 smithi178 ceph-mon[36521]: pgmap v181: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:17:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:13 smithi153 ceph-mon[36228]: pgmap v181: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:17:14.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:13 smithi106 ceph-mon[31965]: pgmap v181: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T02:17:15.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:15 smithi178 ceph-mon[36521]: pgmap v182: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 22 op/s 2024-03-28T02:17:15.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:15 smithi153 ceph-mon[36228]: pgmap v182: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 22 op/s 2024-03-28T02:17:16.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:15 smithi106 ceph-mon[31965]: pgmap v182: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 22 op/s 2024-03-28T02:17:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:17 smithi178 ceph-mon[36521]: pgmap v183: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T02:17:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:17 smithi153 ceph-mon[36228]: pgmap v183: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T02:17:18.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:17 smithi106 ceph-mon[31965]: pgmap v183: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T02:17:18.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:18 smithi153 ceph-mon[36228]: pgmap v184: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T02:17:19.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:18 smithi106 ceph-mon[31965]: pgmap v184: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T02:17:19.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:18 smithi178 ceph-mon[36521]: pgmap v184: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T02:17:21.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:21 smithi106 ceph-mon[31965]: pgmap v185: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T02:17:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:21 smithi178 ceph-mon[36521]: pgmap v185: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T02:17:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:21 smithi153 ceph-mon[36228]: pgmap v185: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T02:17:23.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:23 smithi106 ceph-mon[31965]: pgmap v186: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T02:17:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:23 smithi178 ceph-mon[36521]: pgmap v186: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T02:17:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:23 smithi153 ceph-mon[36228]: pgmap v186: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T02:17:25.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:25 smithi106 ceph-mon[31965]: pgmap v187: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:17:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:25 smithi178 ceph-mon[36521]: pgmap v187: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:17:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:25 smithi153 ceph-mon[36228]: pgmap v187: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:17:27.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:27 smithi106 ceph-mon[31965]: pgmap v188: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:27.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:27 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:17:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:27 smithi178 ceph-mon[36521]: pgmap v188: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:27 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:17:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:27 smithi153 ceph-mon[36228]: pgmap v188: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:27 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:17:28.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:28 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:17:28.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:28 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:17:28.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:28 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:17:28.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:28 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:17:28.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:28 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:17:28.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:28 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:17:28.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:28 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:17:28.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:28 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:17:28.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:28 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:17:29.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:29 smithi106 ceph-mon[31965]: pgmap v189: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:29 smithi178 ceph-mon[36521]: pgmap v189: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:29.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:29 smithi153 ceph-mon[36228]: pgmap v189: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:31.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:31 smithi106 ceph-mon[31965]: pgmap v190: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:31 smithi178 ceph-mon[36521]: pgmap v190: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:31.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:31 smithi153 ceph-mon[36228]: pgmap v190: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:33.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:33 smithi106 ceph-mon[31965]: pgmap v191: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:33 smithi178 ceph-mon[36521]: pgmap v191: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:33.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:33 smithi153 ceph-mon[36228]: pgmap v191: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:35.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:35 smithi106 ceph-mon[31965]: pgmap v192: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:17:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:35 smithi178 ceph-mon[36521]: pgmap v192: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:17:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:35 smithi153 ceph-mon[36228]: pgmap v192: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T02:17:37.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:37 smithi106 ceph-mon[31965]: pgmap v193: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:37 smithi178 ceph-mon[36521]: pgmap v193: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:37.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:37 smithi153 ceph-mon[36228]: pgmap v193: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:39 smithi178 ceph-mon[36521]: pgmap v194: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:39.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:39 smithi153 ceph-mon[36228]: pgmap v194: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:40.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:39 smithi106 ceph-mon[31965]: pgmap v194: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:41 smithi178 ceph-mon[36521]: pgmap v195: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:41.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:41 smithi153 ceph-mon[36228]: pgmap v195: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:42.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:41 smithi106 ceph-mon[31965]: pgmap v195: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:43 smithi178 ceph-mon[36521]: pgmap v196: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:43 smithi153 ceph-mon[36228]: pgmap v196: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:44.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:43 smithi106 ceph-mon[31965]: pgmap v196: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:45 smithi178 ceph-mon[36521]: pgmap v197: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:45.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:45 smithi153 ceph-mon[36228]: pgmap v197: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:46.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:45 smithi106 ceph-mon[31965]: pgmap v197: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:47 smithi178 ceph-mon[36521]: pgmap v198: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:47.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:47 smithi153 ceph-mon[36228]: pgmap v198: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:48.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:47 smithi106 ceph-mon[31965]: pgmap v198: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:49.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:49 smithi178 ceph-mon[36521]: pgmap v199: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:49 smithi153 ceph-mon[36228]: pgmap v199: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:50.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:49 smithi106 ceph-mon[31965]: pgmap v199: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:51 smithi178 ceph-mon[36521]: pgmap v200: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:51.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:51 smithi153 ceph-mon[36228]: pgmap v200: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:52.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:51 smithi106 ceph-mon[31965]: pgmap v200: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:53 smithi178 ceph-mon[36521]: pgmap v201: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:53.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:53 smithi153 ceph-mon[36228]: pgmap v201: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:54.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:53 smithi106 ceph-mon[31965]: pgmap v201: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:55 smithi178 ceph-mon[36521]: pgmap v202: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:55 smithi153 ceph-mon[36228]: pgmap v202: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:56.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:55 smithi106 ceph-mon[31965]: pgmap v202: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:56.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:56 smithi153 ceph-mon[36228]: pgmap v203: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:57.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:56 smithi106 ceph-mon[31965]: pgmap v203: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:57.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:56 smithi178 ceph-mon[36521]: pgmap v203: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:17:59.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:17:59 smithi106 ceph-mon[31965]: pgmap v204: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:17:59 smithi178 ceph-mon[36521]: pgmap v204: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:17:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:17:59 smithi153 ceph-mon[36228]: pgmap v204: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:01.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:01 smithi106 ceph-mon[31965]: pgmap v205: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:01 smithi178 ceph-mon[36521]: pgmap v205: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:01 smithi153 ceph-mon[36228]: pgmap v205: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:03.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:03 smithi106 ceph-mon[31965]: pgmap v206: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:18:03.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:03 smithi178 ceph-mon[36521]: pgmap v206: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:18:03.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:03 smithi153 ceph-mon[36228]: pgmap v206: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:18:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:05 smithi178 ceph-mon[36521]: pgmap v207: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:18:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:05 smithi153 ceph-mon[36228]: pgmap v207: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:18:06.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:05 smithi106 ceph-mon[31965]: pgmap v207: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:18:07.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:07 smithi178 ceph-mon[36521]: pgmap v208: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:18:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:07 smithi153 ceph-mon[36228]: pgmap v208: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:18:08.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:07 smithi106 ceph-mon[31965]: pgmap v208: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:18:09.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:09 smithi178 ceph-mon[36521]: pgmap v209: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:18:09.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:09 smithi178 ceph-mon[36521]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T02:18:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:09 smithi153 ceph-mon[36228]: pgmap v209: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:18:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:09 smithi153 ceph-mon[36228]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T02:18:10.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:09 smithi106 ceph-mon[31965]: pgmap v209: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:18:10.188 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:09 smithi106 ceph-mon[31965]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T02:18:11.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:11 smithi178 ceph-mon[36521]: pgmap v211: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:18:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:11 smithi153 ceph-mon[36228]: pgmap v211: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:18:12.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:11 smithi106 ceph-mon[31965]: pgmap v211: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:18:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:13 smithi178 ceph-mon[36521]: pgmap v212: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:13 smithi153 ceph-mon[36228]: pgmap v212: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:14.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:13 smithi106 ceph-mon[31965]: pgmap v212: 129 pgs: 129 active+clean; 4.4 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:15.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:15 smithi178 ceph-mon[36521]: pgmap v213: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:15.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:15 smithi153 ceph-mon[36228]: pgmap v213: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:16.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:15 smithi106 ceph-mon[31965]: pgmap v213: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:17 smithi178 ceph-mon[36521]: pgmap v214: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:17 smithi153 ceph-mon[36228]: pgmap v214: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:18.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:17 smithi106 ceph-mon[31965]: pgmap v214: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:19.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:19 smithi153 ceph-mon[36228]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T02:18:19.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:19 smithi153 ceph-mon[36228]: pgmap v216: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-03-28T02:18:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:19 smithi106 ceph-mon[31965]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T02:18:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:19 smithi106 ceph-mon[31965]: pgmap v216: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-03-28T02:18:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:19 smithi178 ceph-mon[36521]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T02:18:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:19 smithi178 ceph-mon[36521]: pgmap v216: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-03-28T02:18:21.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:21 smithi106 ceph-mon[31965]: pgmap v217: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:21 smithi178 ceph-mon[36521]: pgmap v217: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:21 smithi153 ceph-mon[36228]: pgmap v217: 129 pgs: 129 active+clean; 4.4 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:18:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:23 smithi178 ceph-mon[36521]: pgmap v218: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:18:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:23 smithi153 ceph-mon[36228]: pgmap v218: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:18:24.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:23 smithi106 ceph-mon[31965]: pgmap v218: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:18:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:25 smithi178 ceph-mon[36521]: pgmap v219: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:18:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:25 smithi153 ceph-mon[36228]: pgmap v219: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:18:26.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:25 smithi106 ceph-mon[31965]: pgmap v219: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:18:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:27 smithi178 ceph-mon[36521]: pgmap v220: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:18:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:27 smithi153 ceph-mon[36228]: pgmap v220: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:18:28.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:27 smithi106 ceph-mon[31965]: pgmap v220: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:18:28.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:28 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:18:28.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:28 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:18:29.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:28 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:18:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:29 smithi178 ceph-mon[36521]: pgmap v221: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T02:18:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:29 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:18:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:29 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:18:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:29 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:18:29.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:29 smithi153 ceph-mon[36228]: pgmap v221: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T02:18:29.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:29 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:18:29.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:29 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:18:29.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:29 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:18:30.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:29 smithi106 ceph-mon[31965]: pgmap v221: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T02:18:30.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:29 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:18:30.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:29 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:18:30.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:29 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:18:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:31 smithi178 ceph-mon[36521]: pgmap v222: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:31.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:31 smithi153 ceph-mon[36228]: pgmap v222: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:32.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:31 smithi106 ceph-mon[31965]: pgmap v222: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:33 smithi178 ceph-mon[36521]: pgmap v223: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:33.884 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:33 smithi106 ceph-mon[31965]: pgmap v223: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:33.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:33 smithi153 ceph-mon[36228]: pgmap v223: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:35 smithi178 ceph-mon[36521]: pgmap v224: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:35 smithi153 ceph-mon[36228]: pgmap v224: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:36.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:35 smithi106 ceph-mon[31965]: pgmap v224: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:37 smithi178 ceph-mon[36521]: pgmap v225: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:37.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:37 smithi153 ceph-mon[36228]: pgmap v225: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:38.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:37 smithi106 ceph-mon[31965]: pgmap v225: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:39.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:39 smithi178 ceph-mon[36521]: pgmap v226: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:39.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:39 smithi153 ceph-mon[36228]: pgmap v226: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:40.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:39 smithi106 ceph-mon[31965]: pgmap v226: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:41 smithi178 ceph-mon[36521]: pgmap v227: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:41.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:41 smithi153 ceph-mon[36228]: pgmap v227: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:42.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:41 smithi106 ceph-mon[31965]: pgmap v227: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:42.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:42 smithi153 ceph-mon[36228]: pgmap v228: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:43.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:42 smithi106 ceph-mon[31965]: pgmap v228: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:43.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:42 smithi178 ceph-mon[36521]: pgmap v228: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:45 smithi178 ceph-mon[36521]: pgmap v229: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:45.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:45 smithi153 ceph-mon[36228]: pgmap v229: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:46.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:45 smithi106 ceph-mon[31965]: pgmap v229: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:47 smithi178 ceph-mon[36521]: pgmap v230: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:47.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:47 smithi153 ceph-mon[36228]: pgmap v230: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:48.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:47 smithi106 ceph-mon[31965]: pgmap v230: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:49.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:49 smithi178 ceph-mon[36521]: pgmap v231: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:49 smithi153 ceph-mon[36228]: pgmap v231: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:50.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:49 smithi106 ceph-mon[31965]: pgmap v231: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:51 smithi178 ceph-mon[36521]: pgmap v232: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:51.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:51 smithi153 ceph-mon[36228]: pgmap v232: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:52.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:51 smithi106 ceph-mon[31965]: pgmap v232: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:53 smithi178 ceph-mon[36521]: pgmap v233: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:53.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:53 smithi153 ceph-mon[36228]: pgmap v233: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:54.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:53 smithi106 ceph-mon[31965]: pgmap v233: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:55 smithi178 ceph-mon[36521]: pgmap v234: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:55.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:55 smithi153 ceph-mon[36228]: pgmap v234: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:56.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:55 smithi106 ceph-mon[31965]: pgmap v234: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:57.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:57 smithi178 ceph-mon[36521]: pgmap v235: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:57 smithi153 ceph-mon[36228]: pgmap v235: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:58.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:57 smithi106 ceph-mon[31965]: pgmap v235: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:18:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:18:59 smithi178 ceph-mon[36521]: pgmap v236: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:18:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:18:59 smithi153 ceph-mon[36228]: pgmap v236: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:00.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:18:59 smithi106 ceph-mon[31965]: pgmap v236: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:01 smithi178 ceph-mon[36521]: pgmap v237: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:01 smithi153 ceph-mon[36228]: pgmap v237: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:02.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:01 smithi106 ceph-mon[31965]: pgmap v237: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:03 smithi178 ceph-mon[36521]: pgmap v238: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:19:03.884 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:03 smithi106 ceph-mon[31965]: pgmap v238: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:19:03.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:03 smithi153 ceph-mon[36228]: pgmap v238: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:19:05.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:05 smithi178 ceph-mon[36521]: pgmap v239: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:19:05.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:05 smithi153 ceph-mon[36228]: pgmap v239: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:19:06.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:05 smithi106 ceph-mon[31965]: pgmap v239: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:19:07.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:07 smithi178 ceph-mon[36521]: pgmap v240: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:19:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:07 smithi153 ceph-mon[36228]: pgmap v240: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:19:08.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:07 smithi106 ceph-mon[31965]: pgmap v240: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:19:08.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:08 smithi178 ceph-mon[36521]: pgmap v241: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:19:08.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:08 smithi153 ceph-mon[36228]: pgmap v241: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:19:09.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:08 smithi106 ceph-mon[31965]: pgmap v241: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:19:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:09 smithi153 ceph-mon[36228]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T02:19:10.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:09 smithi106 ceph-mon[31965]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T02:19:10.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:09 smithi178 ceph-mon[36521]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T02:19:10.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:10 smithi153 ceph-mon[36228]: pgmap v243: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:19:11.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:10 smithi106 ceph-mon[31965]: pgmap v243: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:19:11.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:10 smithi178 ceph-mon[36521]: pgmap v243: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:19:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:13 smithi178 ceph-mon[36521]: pgmap v244: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:13.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:13 smithi153 ceph-mon[36228]: pgmap v244: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:14.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:13 smithi106 ceph-mon[31965]: pgmap v244: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:15.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:15 smithi178 ceph-mon[36521]: pgmap v245: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:15.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:15 smithi153 ceph-mon[36228]: pgmap v245: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:16.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:15 smithi106 ceph-mon[31965]: pgmap v245: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:17.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:17 smithi178 ceph-mon[36521]: pgmap v246: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:17.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:17 smithi153 ceph-mon[36228]: pgmap v246: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:18.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:17 smithi106 ceph-mon[31965]: pgmap v246: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:19.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:19 smithi153 ceph-mon[36228]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T02:19:19.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:19 smithi153 ceph-mon[36228]: pgmap v248: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-03-28T02:19:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:19 smithi106 ceph-mon[31965]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T02:19:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:19 smithi106 ceph-mon[31965]: pgmap v248: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-03-28T02:19:19.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:19 smithi178 ceph-mon[36521]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T02:19:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:19 smithi178 ceph-mon[36521]: pgmap v248: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-03-28T02:19:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:21 smithi178 ceph-mon[36521]: pgmap v249: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:21.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:21 smithi153 ceph-mon[36228]: pgmap v249: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:22.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:21 smithi106 ceph-mon[31965]: pgmap v249: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:19:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:23 smithi178 ceph-mon[36521]: pgmap v250: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:19:23.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:23 smithi153 ceph-mon[36228]: pgmap v250: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:19:24.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:23 smithi106 ceph-mon[31965]: pgmap v250: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:19:25.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:25 smithi178 ceph-mon[36521]: pgmap v251: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:19:25.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:25 smithi153 ceph-mon[36228]: pgmap v251: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:19:26.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:25 smithi106 ceph-mon[31965]: pgmap v251: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:19:27.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:27 smithi178 ceph-mon[36521]: pgmap v252: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:19:27.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:27 smithi153 ceph-mon[36228]: pgmap v252: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:19:28.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:27 smithi106 ceph-mon[31965]: pgmap v252: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:19:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:29 smithi178 ceph-mon[36521]: pgmap v253: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T02:19:29.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:29 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:19:29.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:29 smithi153 ceph-mon[36228]: pgmap v253: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T02:19:29.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:29 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:19:30.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:29 smithi106 ceph-mon[31965]: pgmap v253: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T02:19:30.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:29 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:19:30.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:30 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:19:30.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:30 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:19:30.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:30 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:19:30.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:30 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:19:30.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:30 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:19:30.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:30 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:19:31.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:30 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:19:31.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:30 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:19:31.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:30 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:19:31.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:31 smithi178 ceph-mon[36521]: pgmap v254: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:31.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:31 smithi153 ceph-mon[36228]: pgmap v254: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:32.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:31 smithi106 ceph-mon[31965]: pgmap v254: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:33 smithi178 ceph-mon[36521]: pgmap v255: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:33.884 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:33 smithi106 ceph-mon[31965]: pgmap v255: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:33.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:33 smithi153 ceph-mon[36228]: pgmap v255: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:35 smithi178 ceph-mon[36521]: pgmap v256: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:35.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:35 smithi153 ceph-mon[36228]: pgmap v256: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:36.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:35 smithi106 ceph-mon[31965]: pgmap v256: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:37 smithi178 ceph-mon[36521]: pgmap v257: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:37.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:37 smithi153 ceph-mon[36228]: pgmap v257: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:38.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:37 smithi106 ceph-mon[31965]: pgmap v257: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:38.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:38 smithi153 ceph-mon[36228]: pgmap v258: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:39.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:38 smithi106 ceph-mon[31965]: pgmap v258: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:39.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:38 smithi178 ceph-mon[36521]: pgmap v258: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:41 smithi178 ceph-mon[36521]: pgmap v259: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:41.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:41 smithi153 ceph-mon[36228]: pgmap v259: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:42.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:41 smithi106 ceph-mon[31965]: pgmap v259: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:43 smithi178 ceph-mon[36521]: pgmap v260: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:43.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:43 smithi153 ceph-mon[36228]: pgmap v260: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:44.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:43 smithi106 ceph-mon[31965]: pgmap v260: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:45.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:45 smithi178 ceph-mon[36521]: pgmap v261: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:45.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:45 smithi153 ceph-mon[36228]: pgmap v261: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:46.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:45 smithi106 ceph-mon[31965]: pgmap v261: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:47.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:47 smithi178 ceph-mon[36521]: pgmap v262: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:47.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:47 smithi153 ceph-mon[36228]: pgmap v262: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:48.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:47 smithi106 ceph-mon[31965]: pgmap v262: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:49.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:49 smithi178 ceph-mon[36521]: pgmap v263: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:49 smithi153 ceph-mon[36228]: pgmap v263: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:50.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:49 smithi106 ceph-mon[31965]: pgmap v263: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:51.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:51 smithi178 ceph-mon[36521]: pgmap v264: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:51.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:51 smithi153 ceph-mon[36228]: pgmap v264: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:52.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:51 smithi106 ceph-mon[31965]: pgmap v264: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:53.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:53 smithi178 ceph-mon[36521]: pgmap v265: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:53.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:53 smithi153 ceph-mon[36228]: pgmap v265: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:54.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:53 smithi106 ceph-mon[31965]: pgmap v265: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:55 smithi153 ceph-mon[36228]: pgmap v266: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:56.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:55 smithi106 ceph-mon[31965]: pgmap v266: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:56.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:55 smithi178 ceph-mon[36521]: pgmap v266: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:56.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:56 smithi153 ceph-mon[36228]: pgmap v267: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:57.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:56 smithi106 ceph-mon[31965]: pgmap v267: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:57.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:56 smithi178 ceph-mon[36521]: pgmap v267: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:19:59.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:19:59 smithi178 ceph-mon[36521]: pgmap v268: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:19:59.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:19:59 smithi153 ceph-mon[36228]: pgmap v268: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:00.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:19:59 smithi106 ceph-mon[31965]: pgmap v268: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:00.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:00 smithi178 ceph-mon[36521]: overall HEALTH_OK 2024-03-28T02:20:00.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:00 smithi153 ceph-mon[36228]: overall HEALTH_OK 2024-03-28T02:20:01.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:00 smithi106 ceph-mon[31965]: overall HEALTH_OK 2024-03-28T02:20:01.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:01 smithi178 ceph-mon[36521]: pgmap v269: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:01.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:01 smithi153 ceph-mon[36228]: pgmap v269: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:02.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:01 smithi106 ceph-mon[31965]: pgmap v269: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:03.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:03 smithi178 ceph-mon[36521]: pgmap v270: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:20:03.883 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:03 smithi106 ceph-mon[31965]: pgmap v270: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:20:03.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:03 smithi153 ceph-mon[36228]: pgmap v270: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:20:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:05 smithi153 ceph-mon[36228]: pgmap v271: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:20:06.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:05 smithi106 ceph-mon[31965]: pgmap v271: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:20:06.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:05 smithi178 ceph-mon[36521]: pgmap v271: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:20:06.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:06 smithi153 ceph-mon[36228]: pgmap v272: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:20:07.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:06 smithi106 ceph-mon[31965]: pgmap v272: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:20:07.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:06 smithi178 ceph-mon[36521]: pgmap v272: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:20:09.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:08 smithi106 ceph-mon[31965]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T02:20:09.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:08 smithi106 ceph-mon[31965]: pgmap v274: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:20:09.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:08 smithi178 ceph-mon[36521]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T02:20:09.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:08 smithi178 ceph-mon[36521]: pgmap v274: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:20:09.231 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:08 smithi153 ceph-mon[36228]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T02:20:09.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:08 smithi153 ceph-mon[36228]: pgmap v274: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:20:11.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:11 smithi178 ceph-mon[36521]: pgmap v275: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:20:11.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:11 smithi153 ceph-mon[36228]: pgmap v275: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:20:12.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:11 smithi106 ceph-mon[31965]: pgmap v275: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:20:13.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:13 smithi178 ceph-mon[36521]: pgmap v276: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:13.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:13 smithi153 ceph-mon[36228]: pgmap v276: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:14.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:13 smithi106 ceph-mon[31965]: pgmap v276: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:15.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:15 smithi178 ceph-mon[36521]: pgmap v277: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:15.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:15 smithi153 ceph-mon[36228]: pgmap v277: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:16.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:15 smithi106 ceph-mon[31965]: pgmap v277: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:17.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:17 smithi153 ceph-mon[36228]: pgmap v278: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:18.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:17 smithi106 ceph-mon[31965]: pgmap v278: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:18.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:17 smithi178 ceph-mon[36521]: pgmap v278: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:19.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:19 smithi153 ceph-mon[36228]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T02:20:19.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:19 smithi153 ceph-mon[36228]: pgmap v280: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:19 smithi106 ceph-mon[31965]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T02:20:19.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:19 smithi106 ceph-mon[31965]: pgmap v280: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:19.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:19 smithi178 ceph-mon[36521]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T02:20:19.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:19 smithi178 ceph-mon[36521]: pgmap v280: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:21.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:21 smithi178 ceph-mon[36521]: pgmap v281: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:21.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:21 smithi153 ceph-mon[36228]: pgmap v281: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:22.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:21 smithi106 ceph-mon[31965]: pgmap v281: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:20:23.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:23 smithi178 ceph-mon[36521]: pgmap v282: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:20:23.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:23 smithi153 ceph-mon[36228]: pgmap v282: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:20:24.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:23 smithi106 ceph-mon[31965]: pgmap v282: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:20:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:25 smithi153 ceph-mon[36228]: pgmap v283: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:20:26.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:25 smithi106 ceph-mon[31965]: pgmap v283: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:20:26.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:25 smithi178 ceph-mon[36521]: pgmap v283: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:20:27.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:27 smithi153 ceph-mon[36228]: pgmap v284: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:20:28.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:27 smithi106 ceph-mon[31965]: pgmap v284: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:20:28.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:27 smithi178 ceph-mon[36521]: pgmap v284: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T02:20:29.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:29 smithi153 ceph-mon[36228]: pgmap v285: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T02:20:30.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:29 smithi106 ceph-mon[31965]: pgmap v285: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T02:20:30.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:29 smithi178 ceph-mon[36521]: pgmap v285: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T02:20:30.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:30 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:20:30.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:30 smithi153 ceph-mon[36228]: pgmap v286: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:30.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:30 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:20:30.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:30 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:20:30.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:30 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:20:31.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:30 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:20:31.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:30 smithi106 ceph-mon[31965]: pgmap v286: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:31.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:30 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:20:31.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:30 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:20:31.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:30 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:20:31.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:30 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:20:31.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:30 smithi178 ceph-mon[36521]: pgmap v286: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:31.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:30 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:20:31.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:30 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:20:31.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:30 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:20:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:33 smithi178 ceph-mon[36521]: pgmap v287: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:33.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:33 smithi153 ceph-mon[36228]: pgmap v287: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:34.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:33 smithi106 ceph-mon[31965]: pgmap v287: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:35.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:35 smithi153 ceph-mon[36228]: pgmap v288: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:36.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:35 smithi106 ceph-mon[31965]: pgmap v288: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:36.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:35 smithi178 ceph-mon[36521]: pgmap v288: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:37.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:37 smithi153 ceph-mon[36228]: pgmap v289: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:38.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:37 smithi106 ceph-mon[31965]: pgmap v289: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:38.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:37 smithi178 ceph-mon[36521]: pgmap v289: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:39.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:39 smithi153 ceph-mon[36228]: pgmap v290: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:40.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:39 smithi106 ceph-mon[31965]: pgmap v290: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:40.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:39 smithi178 ceph-mon[36521]: pgmap v290: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:40.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:40 smithi153 ceph-mon[36228]: pgmap v291: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:41.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:40 smithi106 ceph-mon[31965]: pgmap v291: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:41.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:40 smithi178 ceph-mon[36521]: pgmap v291: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:43.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:43 smithi153 ceph-mon[36228]: pgmap v292: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:44.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:43 smithi106 ceph-mon[31965]: pgmap v292: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:44.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:43 smithi178 ceph-mon[36521]: pgmap v292: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:45.923 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:45 smithi178 ceph-mon[36521]: pgmap v293: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:45.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:45 smithi153 ceph-mon[36228]: pgmap v293: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:46.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:45 smithi106 ceph-mon[31965]: pgmap v293: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:47.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:47 smithi153 ceph-mon[36228]: pgmap v294: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:48.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:47 smithi106 ceph-mon[31965]: pgmap v294: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:48.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:47 smithi178 ceph-mon[36521]: pgmap v294: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:48.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:48 smithi153 ceph-mon[36228]: pgmap v295: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:49.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:48 smithi106 ceph-mon[31965]: pgmap v295: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:49.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:48 smithi178 ceph-mon[36521]: pgmap v295: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:51.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:51 smithi153 ceph-mon[36228]: pgmap v296: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:52.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:51 smithi106 ceph-mon[31965]: pgmap v296: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:52.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:51 smithi178 ceph-mon[36521]: pgmap v296: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:53.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:53 smithi153 ceph-mon[36228]: pgmap v297: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:54.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:53 smithi106 ceph-mon[31965]: pgmap v297: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:54.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:53 smithi178 ceph-mon[36521]: pgmap v297: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:20:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:55 smithi153 ceph-mon[36228]: pgmap v298: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:56.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:55 smithi106 ceph-mon[31965]: pgmap v298: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:56.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:55 smithi178 ceph-mon[36521]: pgmap v298: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:56.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:56 smithi153 ceph-mon[36228]: pgmap v299: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:57.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:56 smithi106 ceph-mon[31965]: pgmap v299: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:57.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:56 smithi178 ceph-mon[36521]: pgmap v299: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:20:59.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:20:59 smithi153 ceph-mon[36228]: pgmap v300: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:00.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:20:59 smithi106 ceph-mon[31965]: pgmap v300: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:00.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:20:59 smithi178 ceph-mon[36521]: pgmap v300: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:01 smithi153 ceph-mon[36228]: pgmap v301: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:02.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:01 smithi106 ceph-mon[31965]: pgmap v301: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:02.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:01 smithi178 ceph-mon[36521]: pgmap v301: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:02.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:02 smithi153 ceph-mon[36228]: pgmap v302: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:21:03.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:02 smithi106 ceph-mon[31965]: pgmap v302: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:21:03.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:02 smithi178 ceph-mon[36521]: pgmap v302: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:21:05.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:05 smithi153 ceph-mon[36228]: pgmap v303: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:21:06.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:05 smithi106 ceph-mon[31965]: pgmap v303: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:21:06.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:05 smithi178 ceph-mon[36521]: pgmap v303: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:21:07.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:07 smithi153 ceph-mon[36228]: pgmap v304: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:21:08.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:07 smithi106 ceph-mon[31965]: pgmap v304: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:21:08.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:07 smithi178 ceph-mon[36521]: pgmap v304: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T02:21:08.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:08 smithi153 ceph-mon[36228]: pgmap v305: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:21:09.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:08 smithi106 ceph-mon[31965]: pgmap v305: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:21:09.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:08 smithi178 ceph-mon[36521]: pgmap v305: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T02:21:09.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:09 smithi153 ceph-mon[36228]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T02:21:10.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:09 smithi106 ceph-mon[31965]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T02:21:10.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:09 smithi178 ceph-mon[36521]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T02:21:10.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:10 smithi153 ceph-mon[36228]: pgmap v307: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:21:11.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:10 smithi106 ceph-mon[31965]: pgmap v307: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:21:11.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:10 smithi178 ceph-mon[36521]: pgmap v307: 129 pgs: 129 active+clean; 4.4 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T02:21:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:13 smithi153 ceph-mon[36228]: pgmap v308: 129 pgs: 129 active+clean; 4.4 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:21:13.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:13 smithi153 ceph-mon[36228]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T02:21:14.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:13 smithi106 ceph-mon[31965]: pgmap v308: 129 pgs: 129 active+clean; 4.4 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:21:14.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:13 smithi106 ceph-mon[31965]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T02:21:14.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:13 smithi178 ceph-mon[36521]: pgmap v308: 129 pgs: 129 active+clean; 4.4 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:21:14.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:13 smithi178 ceph-mon[36521]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T02:21:14.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:14 smithi153 ceph-mon[36228]: pgmap v310: 129 pgs: 129 active+clean; 4.4 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:21:15.021 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:14 smithi178 ceph-mon[36521]: pgmap v310: 129 pgs: 129 active+clean; 4.4 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:21:15.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:14 smithi106 ceph-mon[31965]: pgmap v310: 129 pgs: 129 active+clean; 4.4 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T02:21:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:17 smithi153 ceph-mon[36228]: pgmap v311: 129 pgs: 129 active+clean; 4.4 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T02:21:18.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:17 smithi106 ceph-mon[31965]: pgmap v311: 129 pgs: 129 active+clean; 4.4 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T02:21:18.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T02:21:18.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:17 smithi178 ceph-mon[36521]: pgmap v311: 129 pgs: 129 active+clean; 4.4 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T02:21:18.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.214+0000 7f7602019640 1 -- 172.21.15.106:0/3708550880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc107660 msgr2=0x7f75fc107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.214+0000 7f7602019640 1 --2- 172.21.15.106:0/3708550880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc107660 0x7f75fc107a40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009930 tx=0x7f75ec02bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:18.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.214+0000 7f7602019640 1 -- 172.21.15.106:0/3708550880 shutdown_connections 2024-03-28T02:21:18.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.214+0000 7f7602019640 1 --2- 172.21.15.106:0/3708550880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc102010 0x7f75fc10e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.214+0000 7f7602019640 1 --2- 172.21.15.106:0/3708550880 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc101650 0x7f75fc101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.214+0000 7f7602019640 1 --2- 172.21.15.106:0/3708550880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc107660 0x7f75fc107a40 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.214+0000 7f7602019640 1 -- 172.21.15.106:0/3708550880 >> 172.21.15.106:0/3708550880 conn(0x7f75fc0fd160 msgr2=0x7f75fc0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:18.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.214+0000 7f7602019640 1 -- 172.21.15.106:0/3708550880 shutdown_connections 2024-03-28T02:21:18.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.214+0000 7f7602019640 1 -- 172.21.15.106:0/3708550880 wait complete. 2024-03-28T02:21:18.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f7602019640 1 Processor -- start 2024-03-28T02:21:18.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f7602019640 1 -- start start 2024-03-28T02:21:18.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f7602019640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc101650 0x7f75fc197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f7602019640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc102010 0x7f75fc1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f7602019640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc107660 0x7f75fc19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f7602019640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc111a30 con 0x7f75fc101650 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f7602019640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc1118b0 con 0x7f75fc102010 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f7602019640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc111bb0 con 0x7f75fc107660 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f75fb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc101650 0x7f75fc197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f75fb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc101650 0x7f75fc197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55068/0 (socket says 172.21.15.106:55068) 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f75fb7fe640 1 -- 172.21.15.106:0/2202120090 learned_addr learned my addr 172.21.15.106:0/2202120090 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f75faffd640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc102010 0x7f75fc1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.215+0000 7f75fbfff640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc107660 0x7f75fc19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75fb7fe640 1 -- 172.21.15.106:0/2202120090 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc107660 msgr2=0x7f75fc19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75fb7fe640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc107660 0x7f75fc19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75fb7fe640 1 -- 172.21.15.106:0/2202120090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc102010 msgr2=0x7f75fc1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75fb7fe640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc102010 0x7f75fc1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75fb7fe640 1 -- 172.21.15.106:0/2202120090 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc19ce80 con 0x7f75fc101650 2024-03-28T02:21:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75faffd640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc102010 0x7f75fc1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:21:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75fb7fe640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc101650 0x7f75fc197e90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009230 tx=0x7f75ec02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75f8ff9640 1 -- 172.21.15.106:0/2202120090 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec041070 con 0x7f75fc101650 2024-03-28T02:21:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75f8ff9640 1 -- 172.21.15.106:0/2202120090 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75ec004450 con 0x7f75fc101650 2024-03-28T02:21:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75fc19d110 con 0x7f75fc101650 2024-03-28T02:21:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f75f8ff9640 1 -- 172.21.15.106:0/2202120090 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec003e50 con 0x7f75fc101650 2024-03-28T02:21:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.216+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75fc1a4a00 con 0x7f75fc101650 2024-03-28T02:21:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.217+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75fc04f400 con 0x7f75fc101650 2024-03-28T02:21:18.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.218+0000 7f75f8ff9640 1 -- 172.21.15.106:0/2202120090 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f75ec044070 con 0x7f75fc101650 2024-03-28T02:21:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.219+0000 7f75f8ff9640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f75e40a2bb0 0x7f75e40a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.219+0000 7f75faffd640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f75e40a2bb0 0x7f75e40a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.219+0000 7f75f8ff9640 1 -- 172.21.15.106:0/2202120090 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f75ec0f09e0 con 0x7f75fc101650 2024-03-28T02:21:18.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.219+0000 7f75faffd640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f75e40a2bb0 0x7f75e40a5070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0ff6f0 tx=0x7f75f0005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:18.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.224+0000 7f75f8ff9640 1 -- 172.21.15.106:0/2202120090 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f75ec0bce30 con 0x7f75fc101650 2024-03-28T02:21:18.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.419+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 --> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f75fc109c20 con 0x7f75e40a2bb0 2024-03-28T02:21:18.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.425+0000 7f75f8ff9640 1 -- 172.21.15.106:0/2202120090 <== mgr.25315 v2:172.21.15.106:6800/2358705651 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f75fc109c20 con 0x7f75e40a2bb0 2024-03-28T02:21:18.418 INFO:teuthology.orchestra.run.smithi106.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T02:11:17", "first": "2024-03-28T02:12:00", "last": "2024-03-28T02:21:00", "last_pruned": "2024-03-28T02:21:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-28T02:21:18.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.427+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f75e40a2bb0 msgr2=0x7f75e40a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.427+0000 7f7602019640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f75e40a2bb0 0x7f75e40a5070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0ff6f0 tx=0x7f75f0005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.427+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc101650 msgr2=0x7f75fc197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.428+0000 7f7602019640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc101650 0x7f75fc197e90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009230 tx=0x7f75ec02cb30 comp rx=0 tx=0).stop 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.428+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 shutdown_connections 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.428+0000 7f7602019640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f75e40a2bb0 0x7f75e40a5070 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.428+0000 7f7602019640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc107660 0x7f75fc19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.428+0000 7f7602019640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc102010 0x7f75fc1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.428+0000 7f7602019640 1 --2- 172.21.15.106:0/2202120090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75fc101650 0x7f75fc197e90 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.428+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 >> 172.21.15.106:0/2202120090 conn(0x7f75fc0fd160 msgr2=0x7f75fc0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.428+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 shutdown_connections 2024-03-28T02:21:18.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.428+0000 7f7602019640 1 -- 172.21.15.106:0/2202120090 wait complete. 2024-03-28T02:21:18.431 DEBUG:teuthology.orchestra.run.smithi106:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T02:21:18.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3803083898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 msgr2=0x7f84c4107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3803083898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 0x7f84c4107b00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009930 tx=0x7f84c002bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:18.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3803083898 shutdown_connections 2024-03-28T02:21:18.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3803083898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c406a860 0x7f84c410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3803083898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4069ea0 0x7f84c406a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3803083898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 0x7f84c4107b00 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3803083898 >> 172.21.15.106:0/3803083898 conn(0x7f84c4077370 msgr2=0x7f84c4077780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3803083898 shutdown_connections 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3803083898 wait complete. 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.542+0000 7f84cb2dc640 1 Processor -- start 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84cb2dc640 1 -- start start 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84cb2dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4069ea0 0x7f84c419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84cb2dc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c406a860 0x7f84c419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84cb2dc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 0x7f84c419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84cb2dc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4111a80 con 0x7f84c4107720 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84cb2dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4111900 con 0x7f84c4069ea0 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84cb2dc640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4111c00 con 0x7f84c406a860 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9879640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 0x7f84c419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9078640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4069ea0 0x7f84c419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9879640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 0x7f84c419e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55094/0 (socket says 172.21.15.106:55094) 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9879640 1 -- 172.21.15.106:0/3698022800 learned_addr learned my addr 172.21.15.106:0/3698022800 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c8877640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c406a860 0x7f84c419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9879640 1 -- 172.21.15.106:0/3698022800 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c406a860 msgr2=0x7f84c419a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9879640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c406a860 0x7f84c419a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9879640 1 -- 172.21.15.106:0/3698022800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4069ea0 msgr2=0x7f84c419a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9879640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4069ea0 0x7f84c419a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9879640 1 -- 172.21.15.106:0/3698022800 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c419f050 con 0x7f84c4107720 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9078640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4069ea0 0x7f84c419a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84c9879640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 0x7f84c419e920 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00eb10 tx=0x7f84bc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.543+0000 7f84ba7fc640 1 -- 172.21.15.106:0/3698022800 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc004b20 con 0x7f84c4107720 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.544+0000 7f84ba7fc640 1 -- 172.21.15.106:0/3698022800 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84bc004cc0 con 0x7f84c4107720 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.544+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c419f340 con 0x7f84c4107720 2024-03-28T02:21:18.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.544+0000 7f84ba7fc640 1 -- 172.21.15.106:0/3698022800 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc010760 con 0x7f84c4107720 2024-03-28T02:21:18.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.544+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c41a6bd0 con 0x7f84c4107720 2024-03-28T02:21:18.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.544+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c4102da0 con 0x7f84c4107720 2024-03-28T02:21:18.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.545+0000 7f84ba7fc640 1 -- 172.21.15.106:0/3698022800 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f84bc005020 con 0x7f84c4107720 2024-03-28T02:21:18.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.546+0000 7f84ba7fc640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f84b00a2a10 0x7f84b00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.546+0000 7f84c9078640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f84b00a2a10 0x7f84b00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.546+0000 7f84ba7fc640 1 -- 172.21.15.106:0/3698022800 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f84bc0cf900 con 0x7f84c4107720 2024-03-28T02:21:18.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.546+0000 7f84c9078640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f84b00a2a10 0x7f84b00a4ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f84c0005ab0 tx=0x7f84c0000bc0 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:18.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.551+0000 7f84ba7fc640 1 -- 172.21.15.106:0/3698022800 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f84bc09c290 con 0x7f84c4107720 2024-03-28T02:21:18.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.749+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 --> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f84c4061ae0 con 0x7f84b00a2a10 2024-03-28T02:21:18.748 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.756+0000 7f84ba7fc640 1 -- 172.21.15.106:0/3698022800 <== mgr.25315 v2:172.21.15.106:6800/2358705651 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f84c4061ae0 con 0x7f84b00a2a10 2024-03-28T02:21:18.748 INFO:teuthology.orchestra.run.smithi106.stdout:/ 1m 6m3h 2024-03-28T02:21:18.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f84b00a2a10 msgr2=0x7f84b00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f84b00a2a10 0x7f84b00a4ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f84c0005ab0 tx=0x7f84c0000bc0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 msgr2=0x7f84c419e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 0x7f84c419e920 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00eb10 tx=0x7f84bc00c2d0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 shutdown_connections 2024-03-28T02:21:18.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f84b00a2a10 0x7f84b00a4ed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c4107720 0x7f84c419e920 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c406a860 0x7f84c419a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 --2- 172.21.15.106:0/3698022800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4069ea0 0x7f84c419a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 >> 172.21.15.106:0/3698022800 conn(0x7f84c4077370 msgr2=0x7f84c40fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:18.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 shutdown_connections 2024-03-28T02:21:18.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.758+0000 7f84cb2dc640 1 -- 172.21.15.106:0/3698022800 wait complete. 2024-03-28T02:21:18.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:18 smithi106 ceph-mon[31965]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:21:18.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:18 smithi106 ceph-mon[31965]: pgmap v312: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2024-03-28T02:21:18.761 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-28T02:21:18.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:21:18.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.881+0000 7f1c673d4640 1 -- 172.21.15.106:0/2754509119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c601005a0 msgr2=0x7f1c60100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.881+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2754509119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c601005a0 0x7f1c60100a20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009900 tx=0x7f1c5002bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.881+0000 7f1c673d4640 1 -- 172.21.15.106:0/2754509119 shutdown_connections 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.881+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2754509119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c60100f60 0x7f1c6010d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.881+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2754509119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c601005a0 0x7f1c60100a20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.881+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2754509119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c60106730 0x7f1c60106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.881+0000 7f1c673d4640 1 -- 172.21.15.106:0/2754509119 >> 172.21.15.106:0/2754509119 conn(0x7f1c600776a0 msgr2=0x7f1c60077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 -- 172.21.15.106:0/2754509119 shutdown_connections 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 -- 172.21.15.106:0/2754509119 wait complete. 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 Processor -- start 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 -- start start 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c601005a0 0x7f1c60197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c60100f60 0x7f1c60198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c60106730 0x7f1c6019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c601109a0 con 0x7f1c60100f60 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c60110820 con 0x7f1c60106730 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c673d4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c60110b20 con 0x7f1c601005a0 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c6496f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c60100f60 0x7f1c60198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c6496f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c60100f60 0x7f1c60198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55112/0 (socket says 172.21.15.106:55112) 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c6496f640 1 -- 172.21.15.106:0/2184146113 learned_addr learned my addr 172.21.15.106:0/2184146113 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c65170640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c601005a0 0x7f1c60197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.882+0000 7f1c65971640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c60106730 0x7f1c6019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c6496f640 1 -- 172.21.15.106:0/2184146113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c601005a0 msgr2=0x7f1c60197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c6496f640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c601005a0 0x7f1c60197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c6496f640 1 -- 172.21.15.106:0/2184146113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c60106730 msgr2=0x7f1c6019c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c6496f640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c60106730 0x7f1c6019c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c6496f640 1 -- 172.21.15.106:0/2184146113 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6019ced0 con 0x7f1c60100f60 2024-03-28T02:21:18.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c6496f640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c60100f60 0x7f1c60198420 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1c50005940 tx=0x7f1c50004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:18.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c567fc640 1 -- 172.21.15.106:0/2184146113 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50041070 con 0x7f1c60100f60 2024-03-28T02:21:18.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6019d160 con 0x7f1c60100f60 2024-03-28T02:21:18.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c567fc640 1 -- 172.21.15.106:0/2184146113 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c500054e0 con 0x7f1c60100f60 2024-03-28T02:21:18.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c567fc640 1 -- 172.21.15.106:0/2184146113 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50003d00 con 0x7f1c60100f60 2024-03-28T02:21:18.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.883+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c601a4a50 con 0x7f1c60100f60 2024-03-28T02:21:18.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.884+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c60068490 con 0x7f1c60100f60 2024-03-28T02:21:18.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.885+0000 7f1c567fc640 1 -- 172.21.15.106:0/2184146113 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f1c50044070 con 0x7f1c60100f60 2024-03-28T02:21:18.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.885+0000 7f1c567fc640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1c4c0a2970 0x7f1c4c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:18.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.886+0000 7f1c65170640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1c4c0a2970 0x7f1c4c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:18.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.886+0000 7f1c567fc640 1 -- 172.21.15.106:0/2184146113 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f1c500f0fd0 con 0x7f1c60100f60 2024-03-28T02:21:18.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.886+0000 7f1c65170640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1c4c0a2970 0x7f1c4c0a4e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009bd0 tx=0x7f1c5c005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:18.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:18.889+0000 7f1c567fc640 1 -- 172.21.15.106:0/2184146113 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1c500bd420 con 0x7f1c60100f60 2024-03-28T02:21:18.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:18 smithi153 ceph-mon[36228]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:21:18.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:18 smithi153 ceph-mon[36228]: pgmap v312: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2024-03-28T02:21:19.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.066+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c601a4c20 con 0x7f1c60100f60 2024-03-28T02:21:19.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.068+0000 7f1c567fc640 1 -- 172.21.15.106:0/2184146113 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+25414 (secure 0 0 0) 0x7f1c500c14c0 con 0x7f1c60100f60 2024-03-28T02:21:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:21:19.061 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":102,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:21:13.433026+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6801/3501119323":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/2601893696":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/3655727768":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/899620795":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/3501119323":"2024-03-29T02:11:16.390724+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2664246824":"2024-03-29T02:11:16.390724+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1c4c0a2970 msgr2=0x7f1c4c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1c4c0a2970 0x7f1c4c0a4e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009bd0 tx=0x7f1c5c005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c60100f60 msgr2=0x7f1c60198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c60100f60 0x7f1c60198420 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1c50005940 tx=0x7f1c50004270 comp rx=0 tx=0).stop 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 shutdown_connections 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f1c4c0a2970 0x7f1c4c0a4e30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c60106730 0x7f1c6019c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c60100f60 0x7f1c60198420 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 --2- 172.21.15.106:0/2184146113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c601005a0 0x7f1c60197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 >> 172.21.15.106:0/2184146113 conn(0x7f1c600776a0 msgr2=0x7f1c600ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 shutdown_connections 2024-03-28T02:21:19.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.070+0000 7f1c673d4640 1 -- 172.21.15.106:0/2184146113 wait complete. 2024-03-28T02:21:19.073 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T02:08:46.952662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T02:08:47.976273+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T02:08:51.703592+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T02:21:19.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:21:19.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:18 smithi178 ceph-mon[36521]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:21:19.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:18 smithi178 ceph-mon[36521]: pgmap v312: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2024-03-28T02:21:19.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1537627972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101ff0 msgr2=0x7f3f6810e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1537627972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101ff0 0x7f3f6810e500 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009cd0 tx=0x7f3f6002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1537627972 shutdown_connections 2024-03-28T02:21:19.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1537627972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101ff0 0x7f3f6810e500 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1537627972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f68101630 0x7f3f68101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1537627972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f68107640 0x7f3f68107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1537627972 >> 172.21.15.106:0/1537627972 conn(0x7f3f680fd140 msgr2=0x7f3f680ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1537627972 shutdown_connections 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1537627972 wait complete. 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 Processor -- start 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.186+0000 7f3f6eb45640 1 -- start start 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6eb45640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101630 0x7f3f68197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6eb45640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f68101ff0 0x7f3f68198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6eb45640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f68107640 0x7f3f6819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6eb45640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f68111a50 con 0x7f3f68101630 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6eb45640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f681118d0 con 0x7f3f68107640 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6eb45640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f68111bd0 con 0x7f3f68101ff0 2024-03-28T02:21:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6c8e1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101630 0x7f3f68197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6c8e1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101630 0x7f3f68197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55138/0 (socket says 172.21.15.106:55138) 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6c8e1640 1 -- 172.21.15.106:0/1429201050 learned_addr learned my addr 172.21.15.106:0/1429201050 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f67fff640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f68101ff0 0x7f3f68198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6d0e2640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f68107640 0x7f3f6819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6c8e1640 1 -- 172.21.15.106:0/1429201050 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f68101ff0 msgr2=0x7f3f68198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6c8e1640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f68101ff0 0x7f3f68198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6c8e1640 1 -- 172.21.15.106:0/1429201050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f68107640 msgr2=0x7f3f6819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6c8e1640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f68107640 0x7f3f6819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6c8e1640 1 -- 172.21.15.106:0/1429201050 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6819cf20 con 0x7f3f68101630 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6c8e1640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101630 0x7f3f68197f30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00e9f0 tx=0x7f3f5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f65ffb640 1 -- 172.21.15.106:0/1429201050 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c00cd80 con 0x7f3f68101630 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6819d210 con 0x7f3f68101630 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f65ffb640 1 -- 172.21.15.106:0/1429201050 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f5c005500 con 0x7f3f68101630 2024-03-28T02:21:19.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.187+0000 7f3f65ffb640 1 -- 172.21.15.106:0/1429201050 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c010690 con 0x7f3f68101630 2024-03-28T02:21:19.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.188+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f681a4aa0 con 0x7f3f68101630 2024-03-28T02:21:19.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.188+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f68068490 con 0x7f3f68101630 2024-03-28T02:21:19.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.189+0000 7f3f65ffb640 1 -- 172.21.15.106:0/1429201050 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f3f5c0046c0 con 0x7f3f68101630 2024-03-28T02:21:19.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.190+0000 7f3f65ffb640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f580a2970 0x7f3f580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.190+0000 7f3f67fff640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f580a2970 0x7f3f580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.190+0000 7f3f65ffb640 1 -- 172.21.15.106:0/1429201050 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f3f5c0cf5c0 con 0x7f3f68101630 2024-03-28T02:21:19.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.190+0000 7f3f67fff640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f580a2970 0x7f3f580a4e30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f680ff7c0 tx=0x7f3f54005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:19.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.195+0000 7f3f65ffb640 1 -- 172.21.15.106:0/1429201050 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f5c09bf50 con 0x7f3f68101630 2024-03-28T02:21:19.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.369+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f68109bd0 con 0x7f3f68101630 2024-03-28T02:21:19.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.370+0000 7f3f65ffb640 1 -- 172.21.15.106:0/1429201050 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f5c09b850 con 0x7f3f68101630 2024-03-28T02:21:19.362 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:21:19.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f580a2970 msgr2=0x7f3f580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f580a2970 0x7f3f580a4e30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f680ff7c0 tx=0x7f3f54005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101630 msgr2=0x7f3f68197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101630 0x7f3f68197f30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00e9f0 tx=0x7f3f5c00edf0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 shutdown_connections 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f580a2970 0x7f3f580a4e30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f68107640 0x7f3f6819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f68101ff0 0x7f3f68198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 --2- 172.21.15.106:0/1429201050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f68101630 0x7f3f68197f30 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 >> 172.21.15.106:0/1429201050 conn(0x7f3f680fd140 msgr2=0x7f3f680fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 shutdown_connections 2024-03-28T02:21:19.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.372+0000 7f3f6eb45640 1 -- 172.21.15.106:0/1429201050 wait complete. 2024-03-28T02:21:19.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T02:21:19.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.488+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/263488866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d78100bc0 msgr2=0x7f0d78104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.488+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/263488866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d78100bc0 0x7f0d78104960 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009900 tx=0x7f0d6c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:19.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.488+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/263488866 shutdown_connections 2024-03-28T02:21:19.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.488+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/263488866 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d78105090 0x7f0d7810cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.488+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/263488866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d78100bc0 0x7f0d78104960 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.488+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/263488866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d78100210 0x7f0d781005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.488+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/263488866 >> 172.21.15.106:0/263488866 conn(0x7f0d78075800 msgr2=0x7f0d78075c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:19.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.488+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/263488866 shutdown_connections 2024-03-28T02:21:19.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.488+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/263488866 wait complete. 2024-03-28T02:21:19.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7e6ea640 1 Processor -- start 2024-03-28T02:21:19.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7e6ea640 1 -- start start 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7e6ea640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d78100210 0x7f0d78197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7e6ea640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d78105090 0x7f0d78198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7e6ea640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d7819c770 0x7f0d7819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7e6ea640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78111ac0 con 0x7f0d7819c770 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7e6ea640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78111940 con 0x7f0d78105090 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7e6ea640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78111c40 con 0x7f0d78100210 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7cc87640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d7819c770 0x7f0d7819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7cc87640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d7819c770 0x7f0d7819cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55156/0 (socket says 172.21.15.106:55156) 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d7cc87640 1 -- 172.21.15.106:0/3639808703 learned_addr learned my addr 172.21.15.106:0/3639808703 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:19.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d77fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d78100210 0x7f0d78197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.489+0000 7f0d777fe640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d78105090 0x7f0d78198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d7cc87640 1 -- 172.21.15.106:0/3639808703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d78100210 msgr2=0x7f0d78197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d7cc87640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d78100210 0x7f0d78197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d7cc87640 1 -- 172.21.15.106:0/3639808703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d78105090 msgr2=0x7f0d78198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d7cc87640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d78105090 0x7f0d78198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d7cc87640 1 -- 172.21.15.106:0/3639808703 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7819d260 con 0x7f0d7819c770 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d7cc87640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d7819c770 0x7f0d7819cc20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000eb10 tx=0x7f0d7000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d757fa640 1 -- 172.21.15.106:0/3639808703 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d70004b20 con 0x7f0d7819c770 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d757fa640 1 -- 172.21.15.106:0/3639808703 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d70004cc0 con 0x7f0d7819c770 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7819d550 con 0x7f0d7819c770 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d757fa640 1 -- 172.21.15.106:0/3639808703 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d70010760 con 0x7f0d7819c770 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.490+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d781a4d10 con 0x7f0d7819c770 2024-03-28T02:21:19.485 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.491+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d780688e0 con 0x7f0d7819c770 2024-03-28T02:21:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.492+0000 7f0d757fa640 1 -- 172.21.15.106:0/3639808703 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f0d70005020 con 0x7f0d7819c770 2024-03-28T02:21:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.493+0000 7f0d757fa640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f0d600a2970 0x7f0d600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.493+0000 7f0d757fa640 1 -- 172.21.15.106:0/3639808703 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f0d700cf920 con 0x7f0d7819c770 2024-03-28T02:21:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.493+0000 7f0d77fff640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f0d600a2970 0x7f0d600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.493+0000 7f0d77fff640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f0d600a2970 0x7f0d600a4e30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d78076f40 tx=0x7f0d68005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.498+0000 7f0d757fa640 1 -- 172.21.15.106:0/3639808703 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0d7009c2b0 con 0x7f0d7819c770 2024-03-28T02:21:19.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.674+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0d78100bc0 con 0x7f0d7819c770 2024-03-28T02:21:19.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.675+0000 7f0d757fa640 1 -- 172.21.15.106:0/3639808703 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0d7009bbb0 con 0x7f0d7819c770 2024-03-28T02:21:19.667 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:21:19.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f0d600a2970 msgr2=0x7f0d600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f0d600a2970 0x7f0d600a4e30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d78076f40 tx=0x7f0d68005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d7819c770 msgr2=0x7f0d7819cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d7819c770 0x7f0d7819cc20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000eb10 tx=0x7f0d7000c2d0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 shutdown_connections 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f0d600a2970 0x7f0d600a4e30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d7819c770 0x7f0d7819cc20 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d78105090 0x7f0d78198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 --2- 172.21.15.106:0/3639808703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d78100210 0x7f0d78197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 >> 172.21.15.106:0/3639808703 conn(0x7f0d78075800 msgr2=0x7f0d780ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 shutdown_connections 2024-03-28T02:21:19.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.677+0000 7f0d7e6ea640 1 -- 172.21.15.106:0/3639808703 wait complete. 2024-03-28T02:21:19.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T02:21:19.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.792+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3723465364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 msgr2=0x7fa8f410e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.792+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3723465364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 0x7fa8f410e500 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009930 tx=0x7fa8ec02bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:19.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.792+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3723465364 shutdown_connections 2024-03-28T02:21:19.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.792+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3723465364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 0x7fa8f410e500 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.792+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3723465364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f4101630 0x7fa8f4101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.792+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3723465364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8f4107640 0x7fa8f4107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3723465364 >> 172.21.15.106:0/3723465364 conn(0x7fa8f40fd140 msgr2=0x7fa8f40ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3723465364 shutdown_connections 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3723465364 wait complete. 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 Processor -- start 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 -- start start 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8f4101630 0x7fa8f40739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 0x7fa8f4073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f4107640 0x7fa8f4074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4111aa0 con 0x7fa8f4101ff0 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4111920 con 0x7fa8f4107640 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.793+0000 7fa8f94e8640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4111c20 con 0x7fa8f4101630 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f27fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 0x7fa8f4073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f27fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 0x7fa8f4073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55182/0 (socket says 172.21.15.106:55182) 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f27fc640 1 -- 172.21.15.106:0/3679040729 learned_addr learned my addr 172.21.15.106:0/3679040729 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f2ffd640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8f4101630 0x7fa8f40739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f37fe640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f4107640 0x7fa8f4074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f27fc640 1 -- 172.21.15.106:0/3679040729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8f4101630 msgr2=0x7fa8f40739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f27fc640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8f4101630 0x7fa8f40739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f27fc640 1 -- 172.21.15.106:0/3679040729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f4107640 msgr2=0x7fa8f4074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f27fc640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f4107640 0x7fa8f4074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f27fc640 1 -- 172.21.15.106:0/3679040729 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f41a6420 con 0x7fa8f4101ff0 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f27fc640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 0x7fa8f4073ee0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00e9f0 tx=0x7fa8dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8ebfff640 1 -- 172.21.15.106:0/3679040729 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc00cd80 con 0x7fa8f4101ff0 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8ebfff640 1 -- 172.21.15.106:0/3679040729 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8dc010910 con 0x7fa8f4101ff0 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f41a6710 con 0x7fa8f4101ff0 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8ebfff640 1 -- 172.21.15.106:0/3679040729 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc018990 con 0x7fa8f4101ff0 2024-03-28T02:21:19.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.794+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f41a6c60 con 0x7fa8f4101ff0 2024-03-28T02:21:19.790 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.795+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f4068490 con 0x7fa8f4101ff0 2024-03-28T02:21:19.790 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.796+0000 7fa8ebfff640 1 -- 172.21.15.106:0/3679040729 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fa8dc010ac0 con 0x7fa8f4101ff0 2024-03-28T02:21:19.790 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.797+0000 7fa8ebfff640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa8e00a2970 0x7fa8e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:19.790 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.797+0000 7fa8f2ffd640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa8e00a2970 0x7fa8e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:19.790 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.797+0000 7fa8ebfff640 1 -- 172.21.15.106:0/3679040729 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7fa8dc0cf960 con 0x7fa8f4101ff0 2024-03-28T02:21:19.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.797+0000 7fa8f2ffd640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa8e00a2970 0x7fa8e00a4e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4005fd0 tx=0x7fa8e4009500 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:19.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.800+0000 7fa8ebfff640 1 -- 172.21.15.106:0/3679040729 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa8dc09bca0 con 0x7fa8f4101ff0 2024-03-28T02:21:19.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.977+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa8f4109bc0 con 0x7fa8f4101ff0 2024-03-28T02:21:19.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.977+0000 7fa8ebfff640 1 -- 172.21.15.106:0/3679040729 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa8dc09fd40 con 0x7fa8f4101ff0 2024-03-28T02:21:19.970 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 64 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa8e00a2970 msgr2=0x7fa8e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa8e00a2970 0x7fa8e00a4e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4005fd0 tx=0x7fa8e4009500 comp rx=0 tx=0).stop 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 msgr2=0x7fa8f4073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 0x7fa8f4073ee0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00e9f0 tx=0x7fa8dc00eeb0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 shutdown_connections 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa8e00a2970 0x7fa8e00a4e30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f4107640 0x7fa8f4074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8f4101ff0 0x7fa8f4073ee0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 --2- 172.21.15.106:0/3679040729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8f4101630 0x7fa8f40739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 >> 172.21.15.106:0/3679040729 conn(0x7fa8f40fd140 msgr2=0x7fa8f40fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 shutdown_connections 2024-03-28T02:21:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:19.979+0000 7fa8f94e8640 1 -- 172.21.15.106:0/3679040729 wait complete. 2024-03-28T02:21:19.981 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:19 smithi106 ceph-mon[31965]: from='client.15855 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:21:19.981 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:19 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2184146113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:21:19.982 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:19 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1429201050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:21:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:19 smithi153 ceph-mon[36228]: from='client.15855 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:21:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:19 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2184146113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:21:19.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:19 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1429201050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:21:19.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.095+0000 7f868fb44640 1 -- 172.21.15.106:0/3890362459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8688069240 msgr2=0x7f868810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.095+0000 7f868fb44640 1 --2- 172.21.15.106:0/3890362459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8688069240 0x7f868810e4e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8680009cd0 tx=0x7f868002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 -- 172.21.15.106:0/3890362459 shutdown_connections 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 --2- 172.21.15.106:0/3890362459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8688069240 0x7f868810e4e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 --2- 172.21.15.106:0/3890362459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8688068880 0x7f8688068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 --2- 172.21.15.106:0/3890362459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86881076b0 0x7f8688107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 -- 172.21.15.106:0/3890362459 >> 172.21.15.106:0/3890362459 conn(0x7f86880776a0 msgr2=0x7f8688077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 -- 172.21.15.106:0/3890362459 shutdown_connections 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 -- 172.21.15.106:0/3890362459 wait complete. 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 Processor -- start 2024-03-28T02:21:20.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 -- start start 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8688068880 0x7f8688100560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8688069240 0x7f8688100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86881076b0 0x7f8688100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688111a30 con 0x7f8688069240 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86881118b0 con 0x7f86881076b0 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.096+0000 7f868fb44640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688111bb0 con 0x7f8688068880 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d8e0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8688068880 0x7f8688100560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d0df640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8688069240 0x7f8688100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d8e0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8688068880 0x7f8688100560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.106:48514/0 (socket says 172.21.15.106:48514) 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d8e0640 1 -- 172.21.15.106:0/2048864577 learned_addr learned my addr 172.21.15.106:0/2048864577 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868e0e1640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86881076b0 0x7f8688100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d8e0640 1 -- 172.21.15.106:0/2048864577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86881076b0 msgr2=0x7f8688100fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d8e0640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86881076b0 0x7f8688100fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d8e0640 1 -- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8688069240 msgr2=0x7f8688100aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d8e0640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8688069240 0x7f8688100aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d8e0640 1 -- 172.21.15.106:0/2048864577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86881a6310 con 0x7f8688068880 2024-03-28T02:21:20.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d0df640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8688069240 0x7f8688100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:21:20.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868d8e0640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8688068880 0x7f8688100560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f868400e9c0 tx=0x7f868400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:20.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f867effd640 1 -- 172.21.15.106:0/2048864577 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400cde0 con 0x7f8688068880 2024-03-28T02:21:20.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86881a6600 con 0x7f8688068880 2024-03-28T02:21:20.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f867effd640 1 -- 172.21.15.106:0/2048864577 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8684010910 con 0x7f8688068880 2024-03-28T02:21:20.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f867effd640 1 -- 172.21.15.106:0/2048864577 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684018a10 con 0x7f8688068880 2024-03-28T02:21:20.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.097+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86881a6ba0 con 0x7f8688068880 2024-03-28T02:21:20.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.098+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8688068490 con 0x7f8688068880 2024-03-28T02:21:20.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.099+0000 7f867effd640 1 -- 172.21.15.106:0/2048864577 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f868401f020 con 0x7f8688068880 2024-03-28T02:21:20.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.100+0000 7f867effd640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f86740a29a0 0x7f86740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:20.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.100+0000 7f868d0df640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f86740a29a0 0x7f86740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:20.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.100+0000 7f867effd640 1 -- 172.21.15.106:0/2048864577 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f86840cf4e0 con 0x7f8688068880 2024-03-28T02:21:20.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.100+0000 7f868d0df640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f86740a29a0 0x7f86740a4e60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8688101fa0 tx=0x7f8678005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:20.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.104+0000 7f867effd640 1 -- 172.21.15.106:0/2048864577 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f868409be70 con 0x7f8688068880 2024-03-28T02:21:20.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:19 smithi178 ceph-mon[36521]: from='client.15855 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T02:21:20.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:19 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2184146113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:21:20.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:19 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1429201050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:21:20.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.279+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8688109ba0 con 0x7f8688068880 2024-03-28T02:21:20.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.280+0000 7f867effd640 1 -- 172.21.15.106:0/2048864577 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) v1 ==== 103+0+11 (secure 0 0 0) 0x7f868409b770 con 0x7f8688068880 2024-03-28T02:21:20.271 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:21:20.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f86740a29a0 msgr2=0x7f86740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f86740a29a0 0x7f86740a4e60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8688101fa0 tx=0x7f8678005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8688068880 msgr2=0x7f8688100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8688068880 0x7f8688100560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f868400e9c0 tx=0x7f868400ee80 comp rx=0 tx=0).stop 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 shutdown_connections 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f86740a29a0 0x7f86740a4e60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86881076b0 0x7f8688100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8688069240 0x7f8688100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 --2- 172.21.15.106:0/2048864577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8688068880 0x7f8688100560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 >> 172.21.15.106:0/2048864577 conn(0x7f86880776a0 msgr2=0x7f86880ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 shutdown_connections 2024-03-28T02:21:20.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.282+0000 7f868fb44640 1 -- 172.21.15.106:0/2048864577 wait complete. 2024-03-28T02:21:20.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-28T02:21:20.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 -- 172.21.15.106:0/2856319267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101630 msgr2=0x7fbae4101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 --2- 172.21.15.106:0/2856319267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101630 0x7fbae4101ab0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009930 tx=0x7fbacc02bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 -- 172.21.15.106:0/2856319267 shutdown_connections 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 --2- 172.21.15.106:0/2856319267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae4101ff0 0x7fbae410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 --2- 172.21.15.106:0/2856319267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101630 0x7fbae4101ab0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 --2- 172.21.15.106:0/2856319267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbae4107640 0x7fbae4107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 -- 172.21.15.106:0/2856319267 >> 172.21.15.106:0/2856319267 conn(0x7fbae40fd140 msgr2=0x7fbae40ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 -- 172.21.15.106:0/2856319267 shutdown_connections 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 -- 172.21.15.106:0/2856319267 wait complete. 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 Processor -- start 2024-03-28T02:21:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.395+0000 7fbaea481640 1 -- start start 2024-03-28T02:21:20.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbaea481640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae4101630 0x7fbae4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:20.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbaea481640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101ff0 0x7fbae41984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:20.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbaea481640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbae4107640 0x7fbae419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:20.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbaea481640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae4111a50 con 0x7fbae4101ff0 2024-03-28T02:21:20.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbaea481640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae41118d0 con 0x7fbae4101630 2024-03-28T02:21:20.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbaea481640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae4111bd0 con 0x7fbae4107640 2024-03-28T02:21:20.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101ff0 0x7fbae41984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:20.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101ff0 0x7fbae41984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55214/0 (socket says 172.21.15.106:55214) 2024-03-28T02:21:20.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae37fe640 1 -- 172.21.15.106:0/976979716 learned_addr learned my addr 172.21.15.106:0/976979716 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae3fff640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae4101630 0x7fbae4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae8a1e640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbae4107640 0x7fbae419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae37fe640 1 -- 172.21.15.106:0/976979716 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbae4107640 msgr2=0x7fbae419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae37fe640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbae4107640 0x7fbae419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae37fe640 1 -- 172.21.15.106:0/976979716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae4101630 msgr2=0x7fbae4197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae37fe640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae4101630 0x7fbae4197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae37fe640 1 -- 172.21.15.106:0/976979716 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae419cf60 con 0x7fbae4101ff0 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae37fe640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101ff0 0x7fbae41984b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009230 tx=0x7fbacc02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae17fa640 1 -- 172.21.15.106:0/976979716 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbacc041070 con 0x7fbae4101ff0 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae419d1f0 con 0x7fbae4101ff0 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae17fa640 1 -- 172.21.15.106:0/976979716 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbacc004450 con 0x7fbae4101ff0 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbae17fa640 1 -- 172.21.15.106:0/976979716 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbacc003e50 con 0x7fbae4101ff0 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.396+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae41a4ae0 con 0x7fbae4101ff0 2024-03-28T02:21:20.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.397+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae40688e0 con 0x7fbae4101ff0 2024-03-28T02:21:20.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.399+0000 7fbae17fa640 1 -- 172.21.15.106:0/976979716 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fbacc044070 con 0x7fbae4101ff0 2024-03-28T02:21:20.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.400+0000 7fbae17fa640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbad00a2b10 0x7fbad00a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:20.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.400+0000 7fbae3fff640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbad00a2b10 0x7fbad00a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:20.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.400+0000 7fbae17fa640 1 -- 172.21.15.106:0/976979716 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7fbacc0f0da0 con 0x7fbae4101ff0 2024-03-28T02:21:20.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.400+0000 7fbae3fff640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbad00a2b10 0x7fbad00a4fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009b50 tx=0x7fbad4005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:20.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.403+0000 7fbae17fa640 1 -- 172.21.15.106:0/976979716 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbacc0bd1f0 con 0x7fbae4101ff0 2024-03-28T02:21:20.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.626+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fbae4109bf0 con 0x7fbae4101ff0 2024-03-28T02:21:20.626 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.634+0000 7fbae17fa640 1 -- 172.21.15.106:0/976979716 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) v1 ==== 81+0+740373 (secure 0 0 0) 0x7fbacc0c1290 con 0x7fbae4101ff0 2024-03-28T02:21:20.635 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:21:20.635 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:21:20.635 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 25, 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "active_gid": 25315, 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "active_name": "x", 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "active_addrs": { 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6800", 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2358705651 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6801", 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2358705651 2024-03-28T02:21:20.636 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "active_addr": "172.21.15.106:6801/2358705651", 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "active_change": "2024-03-28T02:11:16.390961+0000", 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "active_mgr_features": 4540701547738038271, 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "available": true, 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "standbys": [ 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "gid": 25321, 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "z", 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_features": 4540701547738038271, 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "available_modules": [ 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "alerts", 2024-03-28T02:21:20.637 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.638 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.639 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.640 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.641 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.642 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_destination": { 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_destination", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.643 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Email address to send alerts to", 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_from_name": { 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_from_name", 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.644 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "Ceph", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Email From: name", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_host": { 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_host", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.645 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SMTP server", 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_password": { 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_password", 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.646 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Password to authenticate with", 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_port": { 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_port", 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "465", 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.647 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SMTP port", 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_sender": { 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_sender", 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SMTP envelope sender", 2024-03-28T02:21:20.648 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_ssl": { 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_ssl", 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T02:21:20.649 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_user": { 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_user", 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.650 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "User to authenticate as", 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "balancer", 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "active": { 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "active", 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.651 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "begin_time": { 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "begin_time", 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.652 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0000", 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "begin_weekday": { 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "begin_weekday", 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:20.653 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "0", 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "6", 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_compat_max_iterations": { 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crush_compat_max_iterations", 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.654 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "25", 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "250", 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_compat_metrics": { 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crush_compat_metrics", 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T02:21:20.655 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_compat_step": { 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crush_compat_step", 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0.5", 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "0.001", 2024-03-28T02:21:20.656 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "0.999", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "aggressiveness of optimization", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "end_time": { 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "end_time", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "2359", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T02:21:20.657 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "end_weekday": { 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "end_weekday", 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "0", 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "6", 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T02:21:20.658 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.660 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.661 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.662 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "min_score": { 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "min_score", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:20.663 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "mode": { 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mode", 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.664 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "upmap", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-compat", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "none", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "read", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap-read" 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Balancer mode", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.665 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_ids": { 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pool_ids", 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.666 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.667 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap_max_deviation": { 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "upmap_max_deviation", 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.668 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap_max_optimizations": { 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "upmap_max_optimizations", 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cephadm", 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "agent_down_multiplier": { 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "agent_down_multiplier", 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "3.0", 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.670 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "agent_refresh_rate": { 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "agent_refresh_rate", 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "20", 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T02:21:20.671 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "agent_starting_port": { 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "agent_starting_port", 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "4721", 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T02:21:20.672 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_ptrace": { 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "allow_ptrace", 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.673 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "autotune_interval": { 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "autotune_interval", 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.674 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "autotune_memory_target_ratio": { 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0.7", 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.675 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm_log_destination": { 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cephadm_log_destination", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "file", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "file,syslog", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "syslog" 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.676 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "cgroups_split": { 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cgroups_split", 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.677 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "config_checks_enabled": { 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "config_checks_enabled", 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "config_dashboard": { 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "config_dashboard", 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.679 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_alertmanager": { 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_alertmanager", 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_base": { 2024-03-28T02:21:20.680 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_base", 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Container image name, without the tag", 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_elasticsearch": { 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_elasticsearch", 2024-03-28T02:21:20.681 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "elasticsearch container image", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_grafana": { 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_grafana", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_haproxy": { 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_haproxy", 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.683 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "HAproxy container image", 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_jaeger_agent": { 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_jaeger_agent", 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T02:21:20.684 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Jaeger agent container image", 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_jaeger_collector": { 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_jaeger_collector", 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T02:21:20.685 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Jaeger collector container image", 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_jaeger_query": { 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_jaeger_query", 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.686 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Jaeger query container image", 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_keepalived": { 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_keepalived", 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.687 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Keepalived container image", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_loki": { 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_loki", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Loki container image", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.688 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_node_exporter": { 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_node_exporter", 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_nvmeof": { 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_nvmeof", 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Nvme-of container image", 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_prometheus": { 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_prometheus", 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.691 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_promtail": { 2024-03-28T02:21:20.693 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_promtail", 2024-03-28T02:21:20.693 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.693 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.693 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.693 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Promtail container image", 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_snmp_gateway": { 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_snmp_gateway", 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T02:21:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SNMP Gateway container image", 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "container_init": { 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_init", 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "daemon_cache_timeout": { 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "daemon_cache_timeout", 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.696 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "default_cephadm_command_timeout": { 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "900", 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T02:21:20.697 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "default_registry": { 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "default_registry", 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io", 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.698 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "device_cache_timeout": { 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "device_cache_timeout", 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "1800", 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "seconds to cache device inventory", 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.699 INFO:teuthology.orchestra.run.smithi106.stdout: "device_enhanced_scan": { 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "device_enhanced_scan", 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "facts_cache_timeout": { 2024-03-28T02:21:20.700 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "facts_cache_timeout", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "seconds to cache host facts data", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "host_check_interval": { 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "host_check_interval", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.701 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to perform a host check", 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "hw_monitoring": { 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "hw_monitoring", 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "inventory_list_all": { 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "inventory_list_all", 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.703 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.704 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "log_refresh_metadata": { 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_refresh_metadata", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.705 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.706 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.707 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.708 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.709 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.710 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "*", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "max_count_per_host": { 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_count_per_host", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.711 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "max number of daemons per service per host", 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd_draining_count": { 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_osd_draining_count", 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.712 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "migration_current": { 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "migration_current", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "internal - do not modify", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.713 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "mode": { 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mode", 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "root", 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm-package", 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "root" 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T02:21:20.714 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "oob_default_addr": { 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "oob_default_addr", 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "169.254.1.1", 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "prometheus_alerts_path": { 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "prometheus_alerts_path", 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_insecure": { 2024-03-28T02:21:20.716 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_insecure", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_password": { 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_password", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.717 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_url": { 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_url", 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.718 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_username": { 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_username", 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "secure_monitoring_stack": { 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "secure_monitoring_stack", 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.720 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "service_discovery_port": { 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "service_discovery_port", 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8765", 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.721 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "cephadm service discovery port", 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "ssh_config_file": { 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssh_config_file", 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.722 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "use_agent": { 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "use_agent", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.723 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "use_repo_digest": { 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "use_repo_digest", 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.724 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_on_failed_host_check": { 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_on_failed_host_check", 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_on_stray_daemons": { 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_on_stray_daemons", 2024-03-28T02:21:20.725 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_on_stray_hosts": { 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_on_stray_hosts", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.726 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crash", 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.727 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.728 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.729 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.730 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.731 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "retain_interval": { 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "retain_interval", 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "31536000", 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T02:21:20.732 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_recent_interval": { 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_recent_interval", 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "1209600", 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.733 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "dashboard", 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:20.734 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.735 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.736 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "AUDIT_API_ENABLED": { 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.737 INFO:teuthology.orchestra.run.smithi106.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T02:21:20.738 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.739 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.740 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.741 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.742 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.743 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.744 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.745 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.746 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "admin", 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_URL": { 2024-03-28T02:21:20.748 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_URL", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "admin", 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.750 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.751 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.752 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.753 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "{}", 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T02:21:20.755 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.756 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.757 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.758 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.759 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.760 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.761 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.762 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T02:21:20.763 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.764 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8", 2024-03-28T02:21:20.765 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "45", 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.766 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.767 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "admin", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.768 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.769 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T02:21:20.770 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.771 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.772 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.773 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "cross_origin_url": { 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cross_origin_url", 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.774 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "crt_file": { 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crt_file", 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "debug": { 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "debug", 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Enable/disable debug options", 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.776 INFO:teuthology.orchestra.run.smithi106.stdout: "jwt_token_ttl": { 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "jwt_token_ttl", 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "28800", 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "key_file": { 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "key_file", 2024-03-28T02:21:20.777 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.778 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.780 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.781 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.782 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "motd": { 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "motd", 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "The message of the day", 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "redirect_resolve_ip_addr": { 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "server_addr": { 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_addr", 2024-03-28T02:21:20.784 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "::", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "server_port": { 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_port", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.785 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8080", 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "ssl": { 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssl", 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.786 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "ssl_server_port": { 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssl_server_port", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8443", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.787 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_behaviour": { 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_behaviour", 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "redirect", 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.788 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "redirect" 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_error_status_code": { 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_error_status_code", 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "500", 2024-03-28T02:21:20.789 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "400", 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "599", 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "url_prefix": { 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "url_prefix", 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.790 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "devicehealth", 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "enable_monitoring": { 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enable_monitoring", 2024-03-28T02:21:20.791 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "monitor device health metrics", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.792 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.794 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.795 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.796 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mark_out_threshold": { 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mark_out_threshold", 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "2419200", 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T02:21:20.797 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": { 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pool_name", 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "device_health_metrics", 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.798 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "retention_period": { 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "retention_period", 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "15552000", 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how long to retain device health metrics", 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.799 INFO:teuthology.orchestra.run.smithi106.stdout: "scrape_frequency": { 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "scrape_frequency", 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "86400", 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "self_heal": { 2024-03-28T02:21:20.800 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "self_heal", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.801 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_threshold": { 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_threshold", 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "7257600", 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "diskprediction_local", 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.803 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.804 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.805 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.806 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.807 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "predict_interval": { 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "predict_interval", 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "86400", 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.808 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "predictor_model": { 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "predictor_model", 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "prophetstor", 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.809 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.810 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "influx", 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": false, 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "influxdb python module not found", 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "batch_size": { 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "batch_size", 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5000", 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.811 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "database": { 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "database", 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "ceph", 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T02:21:20.812 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "hostname": { 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "hostname", 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "InfluxDB server hostname", 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.813 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "30", 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "5", 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.815 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.816 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.817 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.818 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "password": { 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "password", 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.819 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "password of InfluxDB server user", 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "port": { 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "port", 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8086", 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "InfluxDB server port", 2024-03-28T02:21:20.820 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "ssl": { 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssl", 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "false", 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.821 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "threads": { 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "threads", 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "32", 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.822 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "username": { 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "username", 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "username of InfluxDB server user", 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.823 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "verify_ssl": { 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "verify_ssl", 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "true", 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.824 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "insights", 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.825 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.826 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.827 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.828 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "iostat", 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.830 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.831 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.832 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.833 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "k8sevents", 2024-03-28T02:21:20.834 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_event_retention_days": { 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ceph_event_retention_days", 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "7", 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.835 INFO:teuthology.orchestra.run.smithi106.stdout: "config_check_secs": { 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "config_check_secs", 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "10", 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.836 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.837 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.838 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.839 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.840 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "localpool", 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "failure_domain": { 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "failure_domain", 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.841 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "host", 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "failure domain for any created local pool", 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.843 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.845 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": { 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "min_size", 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "default min_size for any created local pool", 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "num_rep": { 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "num_rep", 2024-03-28T02:21:20.847 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "3", 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "default replica count for any created local pool", 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": { 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pg_num", 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.848 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "128", 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "prefix": { 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "prefix", 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.849 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "name prefix for any created local pool", 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "subtree": { 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "subtree", 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "rack", 2024-03-28T02:21:20.850 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mds_autoscaler", 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.851 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.852 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.853 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.854 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.855 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mirroring", 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.856 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.858 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.859 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.860 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "nfs", 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.861 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.862 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.863 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "orchestrator", 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "fail_fs": { 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "fail_fs", 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T02:21:20.866 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.867 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.868 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.869 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.870 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "orchestrator": { 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "orchestrator", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.871 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm", 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rook", 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "test_orchestrator" 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Orchestrator backend", 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "osd_perf_query", 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.872 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.873 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.874 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.875 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.876 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "osd_support", 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.877 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.878 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.879 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.880 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.881 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pg_autoscaler", 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.882 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.884 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.885 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.886 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "threshold": { 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "threshold", 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "3.0", 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1.0", 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "scaling threshold", 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "progress", 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_pg_recovery_event": { 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "allow_pg_recovery_event", 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T02:21:20.889 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "enabled": { 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enabled", 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.890 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.891 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.892 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.893 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.894 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.895 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.895 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.895 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.895 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.895 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.895 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.895 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.897 INFO:teuthology.orchestra.run.smithi106.stdout: "max_completed_events": { 2024-03-28T02:21:20.897 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_completed_events", 2024-03-28T02:21:20.897 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.897 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "50", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "number of past completed events to remember", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:20.898 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how long the module is going to sleep", 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "prometheus", 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.899 INFO:teuthology.orchestra.run.smithi106.stdout: "cache": { 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cache", 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "exclude_perf_counters": { 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "exclude_perf_counters", 2024-03-28T02:21:20.900 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.901 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.902 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.904 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.905 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_stats_pools": { 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_stats_pools", 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "300", 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "scrape_interval": { 2024-03-28T02:21:20.907 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "scrape_interval", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "15.0", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "server_addr": { 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_addr", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.908 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "::", 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "server_port": { 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_port", 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.909 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "9283", 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "stale_cache_strategy": { 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "stale_cache_strategy", 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "log", 2024-03-28T02:21:20.910 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_behaviour": { 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_behaviour", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "default", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.911 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "default", 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "error" 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_error_status_code": { 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_error_status_code", 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "500", 2024-03-28T02:21:20.912 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "400", 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "599", 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_support", 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.913 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.914 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.915 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.915 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.915 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.915 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.915 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.915 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.915 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.916 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.917 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "max_concurrent_snap_create": { 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_concurrent_snap_create", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.918 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "mirror_snapshot_schedule": { 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.919 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "trash_purge_schedule": { 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "trash_purge_schedule", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.920 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "restful", 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "enable_auth": { 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enable_auth", 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.921 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "key_file": { 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "key_file", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.922 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.923 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.924 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.925 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.926 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.927 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "server_addr": { 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_addr", 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "server_port": { 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_port", 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rgw", 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.930 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.931 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.932 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.933 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rook", 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.934 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.935 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.936 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.937 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.938 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "storage_class": { 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "storage_class", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "local", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.939 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "selftest", 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.940 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.941 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.943 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "roption1": { 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "roption1", 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.945 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "roption2": { 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "roption2", 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "xyz", 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.946 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption1": { 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption1", 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption2": { 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption2", 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption3": { 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption3", 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption4": { 2024-03-28T02:21:20.949 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption4", 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption5": { 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption5", 2024-03-28T02:21:20.950 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption6": { 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption6", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.951 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption7": { 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption7", 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.952 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "42", 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "testkey": { 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "testkey", 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.953 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "testlkey": { 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "testlkey", 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.954 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "testnewline": { 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "testnewline", 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.955 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snap_schedule", 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_m_granularity": { 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "allow_m_granularity", 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.956 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "dump_on_update": { 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "dump_on_update", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.957 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "dump database to debug log on update", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.958 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.959 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.960 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.961 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "stats", 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.962 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.963 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.964 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.965 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.966 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "status", 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.967 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.968 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.970 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.971 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "telegraf", 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "address": { 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "address", 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T02:21:20.972 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "15", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.973 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.974 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.975 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.976 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.977 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "telemetry", 2024-03-28T02:21:20.978 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_basic": { 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_basic", 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.979 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_crash": { 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_crash", 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_device": { 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_device", 2024-03-28T02:21:20.980 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_ident": { 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_ident", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.981 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_perf": { 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_perf", 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.982 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "contact": { 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "contact", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.983 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "description": { 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "description", 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.984 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "device_url": { 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "device_url", 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.985 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "enabled": { 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enabled", 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.986 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "24", 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "8", 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "last_opt_revision": { 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "last_opt_revision", 2024-03-28T02:21:20.987 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "1", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "leaderboard": { 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "leaderboard", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.988 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "leaderboard_description": { 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "leaderboard_description", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.989 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.990 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.991 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.992 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.994 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "organization": { 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "organization", 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.995 INFO:teuthology.orchestra.run.smithi106.stdout: "proxy": { 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "proxy", 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "url": { 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "url", 2024-03-28T02:21:20.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "test_orchestrator", 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:20.998 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:20.999 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.000 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.001 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.002 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "volumes", 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.003 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.004 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.005 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.006 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.007 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "max_concurrent_clones": { 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_concurrent_clones", 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "4", 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "periodic_async_work": { 2024-03-28T02:21:21.009 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "periodic_async_work", 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Periodically check for async work", 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "snapshot_clone_delay": { 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snapshot_clone_delay", 2024-03-28T02:21:21.010 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "snapshot_clone_no_wait": { 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.011 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix", 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.012 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "discovery_interval": { 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "discovery_interval", 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "100", 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.013 INFO:teuthology.orchestra.run.smithi106.stdout: "identifier": { 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "identifier", 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.014 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.015 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.016 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.017 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.018 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.019 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "zabbix_host": { 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix_host", 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.020 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "zabbix_port": { 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix_port", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10051", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.021 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "zabbix_sender": { 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix_sender", 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.022 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: "gid": 25391, 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "y", 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_features": 4540701547738038271, 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: "available_modules": [ 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "alerts", 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.023 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.025 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.026 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.027 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.028 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_destination": { 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_destination", 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.029 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Email address to send alerts to", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_from_name": { 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_from_name", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "Ceph", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.030 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Email From: name", 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_host": { 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_host", 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SMTP server", 2024-03-28T02:21:21.031 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_password": { 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_password", 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Password to authenticate with", 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.032 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_port": { 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_port", 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "465", 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SMTP port", 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_sender": { 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_sender", 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SMTP envelope sender", 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_ssl": { 2024-03-28T02:21:21.034 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_ssl", 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_user": { 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_user", 2024-03-28T02:21:21.035 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "User to authenticate as", 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.036 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "balancer", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "active": { 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "active", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.037 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "begin_time": { 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "begin_time", 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0000", 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.038 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "begin_weekday": { 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "begin_weekday", 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "0", 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "6", 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_compat_max_iterations": { 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crush_compat_max_iterations", 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "25", 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "250", 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.040 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_compat_metrics": { 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crush_compat_metrics", 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_compat_step": { 2024-03-28T02:21:21.041 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crush_compat_step", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0.5", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "0.001", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "0.999", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "aggressiveness of optimization", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "end_time": { 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "end_time", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.042 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "2359", 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "end_weekday": { 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "end_weekday", 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "0", 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "6", 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.044 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.045 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.046 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.047 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.048 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "min_score": { 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "min_score", 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "mode": { 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mode", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "upmap", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-compat", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "none", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "read", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap", 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap-read" 2024-03-28T02:21:21.050 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Balancer mode", 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_ids": { 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pool_ids", 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T02:21:21.051 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.052 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap_max_deviation": { 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "upmap_max_deviation", 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap_max_optimizations": { 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "upmap_max_optimizations", 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.054 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cephadm", 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "agent_down_multiplier": { 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "agent_down_multiplier", 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "3.0", 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.055 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "agent_refresh_rate": { 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "agent_refresh_rate", 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "20", 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T02:21:21.056 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "agent_starting_port": { 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "agent_starting_port", 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "4721", 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.057 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_ptrace": { 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "allow_ptrace", 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.058 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "autotune_interval": { 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "autotune_interval", 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "autotune_memory_target_ratio": { 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0.7", 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm_log_destination": { 2024-03-28T02:21:21.060 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cephadm_log_destination", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "file", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "file,syslog", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "syslog" 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "cgroups_split": { 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cgroups_split", 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.062 INFO:teuthology.orchestra.run.smithi106.stdout: "config_checks_enabled": { 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "config_checks_enabled", 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "config_dashboard": { 2024-03-28T02:21:21.063 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "config_dashboard", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_alertmanager": { 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_alertmanager", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.064 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_base": { 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_base", 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.065 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Container image name, without the tag", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_elasticsearch": { 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_elasticsearch", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.066 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "elasticsearch container image", 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_grafana": { 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_grafana", 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.067 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_haproxy": { 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_haproxy", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "HAproxy container image", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.068 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_jaeger_agent": { 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_jaeger_agent", 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Jaeger agent container image", 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.069 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_jaeger_collector": { 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_jaeger_collector", 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Jaeger collector container image", 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_jaeger_query": { 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_jaeger_query", 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Jaeger query container image", 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.071 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_keepalived": { 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_keepalived", 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Keepalived container image", 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_loki": { 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_loki", 2024-03-28T02:21:21.072 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Loki container image", 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_node_exporter": { 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_node_exporter", 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.073 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_nvmeof": { 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_nvmeof", 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.074 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Nvme-of container image", 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_prometheus": { 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_prometheus", 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.075 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_promtail": { 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_promtail", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.076 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Promtail container image", 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_snmp_gateway": { 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_snmp_gateway", 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.077 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SNMP Gateway container image", 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "container_init": { 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_init", 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T02:21:21.078 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "daemon_cache_timeout": { 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "daemon_cache_timeout", 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.079 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "default_cephadm_command_timeout": { 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "900", 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "default_registry": { 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "default_registry", 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io", 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.081 INFO:teuthology.orchestra.run.smithi106.stdout: "device_cache_timeout": { 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "device_cache_timeout", 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "1800", 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "seconds to cache device inventory", 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "device_enhanced_scan": { 2024-03-28T02:21:21.082 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "device_enhanced_scan", 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "facts_cache_timeout": { 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "facts_cache_timeout", 2024-03-28T02:21:21.083 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "seconds to cache host facts data", 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "host_check_interval": { 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "host_check_interval", 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.084 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to perform a host check", 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "hw_monitoring": { 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "hw_monitoring", 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.085 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "inventory_list_all": { 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "inventory_list_all", 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.086 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.087 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.088 INFO:teuthology.orchestra.run.smithi106.stdout: "log_refresh_metadata": { 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_refresh_metadata", 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.089 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.090 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.091 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.092 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "*", 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.094 INFO:teuthology.orchestra.run.smithi106.stdout: "max_count_per_host": { 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_count_per_host", 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "max number of daemons per service per host", 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd_draining_count": { 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_osd_draining_count", 2024-03-28T02:21:21.095 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "migration_current": { 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "migration_current", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.096 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "internal - do not modify", 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "mode": { 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mode", 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.097 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "root", 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm-package", 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "root" 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "oob_default_addr": { 2024-03-28T02:21:21.098 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "oob_default_addr", 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "169.254.1.1", 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "prometheus_alerts_path": { 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "prometheus_alerts_path", 2024-03-28T02:21:21.099 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_insecure": { 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_insecure", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.100 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_password": { 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_password", 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.101 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_url": { 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_url", 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.102 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_username": { 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_username", 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.103 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "secure_monitoring_stack": { 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "secure_monitoring_stack", 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "service_discovery_port": { 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "service_discovery_port", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8765", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "cephadm service discovery port", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "ssh_config_file": { 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssh_config_file", 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.106 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "use_agent": { 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "use_agent", 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.107 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "use_repo_digest": { 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "use_repo_digest", 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.108 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_on_failed_host_check": { 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_on_failed_host_check", 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_on_stray_daemons": { 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_on_stray_daemons", 2024-03-28T02:21:21.109 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_on_stray_hosts": { 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_on_stray_hosts", 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.110 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crash", 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.111 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.112 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.113 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.114 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.115 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "retain_interval": { 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "retain_interval", 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "31536000", 2024-03-28T02:21:21.116 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_recent_interval": { 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_recent_interval", 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "1209600", 2024-03-28T02:21:21.117 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "dashboard", 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.118 INFO:teuthology.orchestra.run.smithi106.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T02:21:21.119 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.120 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "AUDIT_API_ENABLED": { 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.121 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.122 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.123 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.124 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.125 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.128 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T02:21:21.129 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T02:21:21.130 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T02:21:21.131 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "admin", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.132 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_URL": { 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_URL", 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.133 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "admin", 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.134 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.135 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.136 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T02:21:21.138 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "{}", 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T02:21:21.139 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.140 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.141 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.142 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.143 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.144 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.145 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.146 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.147 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.148 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8", 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.151 INFO:teuthology.orchestra.run.smithi106.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "45", 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T02:21:21.152 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.153 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "admin", 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.154 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.155 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.156 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.157 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.158 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.159 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "cross_origin_url": { 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cross_origin_url", 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.160 INFO:teuthology.orchestra.run.smithi106.stdout: "crt_file": { 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crt_file", 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "debug": { 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "debug", 2024-03-28T02:21:21.161 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Enable/disable debug options", 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "jwt_token_ttl": { 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "jwt_token_ttl", 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.162 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "28800", 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "key_file": { 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "key_file", 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.163 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.164 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.165 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.166 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.167 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.168 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "motd": { 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "motd", 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "The message of the day", 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.169 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "redirect_resolve_ip_addr": { 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.170 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "server_addr": { 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_addr", 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "::", 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.171 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "server_port": { 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_port", 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8080", 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.172 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "ssl": { 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssl", 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "ssl_server_port": { 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssl_server_port", 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8443", 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.174 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_behaviour": { 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_behaviour", 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "redirect", 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "redirect" 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_error_status_code": { 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_error_status_code", 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "500", 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "400", 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "599", 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.176 INFO:teuthology.orchestra.run.smithi106.stdout: "url_prefix": { 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "url_prefix", 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "devicehealth", 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "enable_monitoring": { 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enable_monitoring", 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.178 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "monitor device health metrics", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.179 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.180 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.181 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.182 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "mark_out_threshold": { 2024-03-28T02:21:21.183 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mark_out_threshold", 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "2419200", 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": { 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pool_name", 2024-03-28T02:21:21.184 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "device_health_metrics", 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "retention_period": { 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "retention_period", 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.185 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "15552000", 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how long to retain device health metrics", 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "scrape_frequency": { 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "scrape_frequency", 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.186 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "86400", 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "self_heal": { 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "self_heal", 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.187 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.188 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_threshold": { 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_threshold", 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "7257600", 2024-03-28T02:21:21.189 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "diskprediction_local", 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.190 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.191 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.192 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.193 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.194 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "predict_interval": { 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "predict_interval", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "86400", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.195 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "predictor_model": { 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "predictor_model", 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "prophetstor", 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.196 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.197 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "influx", 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": false, 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "influxdb python module not found", 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "batch_size": { 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "batch_size", 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.198 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5000", 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "database": { 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "database", 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "ceph", 2024-03-28T02:21:21.199 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "hostname": { 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "hostname", 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.200 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "InfluxDB server hostname", 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "30", 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "5", 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.201 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.202 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.203 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.204 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.205 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "password": { 2024-03-28T02:21:21.206 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "password", 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "password of InfluxDB server user", 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "port": { 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "port", 2024-03-28T02:21:21.207 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8086", 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "InfluxDB server port", 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "ssl": { 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssl", 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.208 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "false", 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "threads": { 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "threads", 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.209 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "32", 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "username": { 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "username", 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.210 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "username of InfluxDB server user", 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "verify_ssl": { 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "verify_ssl", 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "true", 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.211 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "insights", 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.212 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.213 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.214 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.215 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.216 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "iostat", 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.217 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.218 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.220 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.221 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "k8sevents", 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_event_retention_days": { 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ceph_event_retention_days", 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.222 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "7", 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "config_check_secs": { 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "config_check_secs", 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "10", 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.224 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.225 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.226 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.227 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.228 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "localpool", 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "failure_domain": { 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "failure_domain", 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "host", 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.229 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "failure domain for any created local pool", 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.230 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.231 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.232 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.233 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": { 2024-03-28T02:21:21.234 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "min_size", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "default min_size for any created local pool", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "num_rep": { 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "num_rep", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.235 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "3", 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "default replica count for any created local pool", 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": { 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pg_num", 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.236 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "128", 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "prefix": { 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "prefix", 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.237 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "name prefix for any created local pool", 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "subtree": { 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "subtree", 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.238 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "rack", 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mds_autoscaler", 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.239 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.240 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.241 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mirroring", 2024-03-28T02:21:21.244 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.245 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.246 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.248 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "nfs", 2024-03-28T02:21:21.249 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.250 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.251 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.252 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.253 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "orchestrator", 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "fail_fs": { 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "fail_fs", 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.255 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.256 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.257 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.259 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "orchestrator": { 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "orchestrator", 2024-03-28T02:21:21.260 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm", 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "rook", 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "test_orchestrator" 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Orchestrator backend", 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "osd_perf_query", 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.266 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "osd_support", 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.267 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.268 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.269 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.270 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.271 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pg_autoscaler", 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.272 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.273 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.274 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.275 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.276 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "threshold": { 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "threshold", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.277 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "3.0", 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1.0", 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "scaling threshold", 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "progress", 2024-03-28T02:21:21.278 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_pg_recovery_event": { 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "allow_pg_recovery_event", 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.279 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "enabled": { 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enabled", 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.280 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.281 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.282 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.283 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "max_completed_events": { 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_completed_events", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "50", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.285 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "number of past completed events to remember", 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.286 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how long the module is going to sleep", 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "prometheus", 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "cache": { 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cache", 2024-03-28T02:21:21.287 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "exclude_perf_counters": { 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "exclude_perf_counters", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.288 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.289 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.291 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.292 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.293 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_stats_pools": { 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_stats_pools", 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "300", 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "scrape_interval": { 2024-03-28T02:21:21.295 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "scrape_interval", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "15.0", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "server_addr": { 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_addr", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.296 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "::", 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "server_port": { 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_port", 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.297 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "9283", 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "stale_cache_strategy": { 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "stale_cache_strategy", 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.298 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.299 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.299 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "log", 2024-03-28T02:21:21.299 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.299 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.299 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.299 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.299 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.299 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.300 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.300 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_behaviour": { 2024-03-28T02:21:21.300 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_behaviour", 2024-03-28T02:21:21.300 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.300 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.300 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.300 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "default", 2024-03-28T02:21:21.301 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.301 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.301 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.301 INFO:teuthology.orchestra.run.smithi106.stdout: "default", 2024-03-28T02:21:21.301 INFO:teuthology.orchestra.run.smithi106.stdout: "error" 2024-03-28T02:21:21.301 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.301 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.302 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.302 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.302 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.302 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_error_status_code": { 2024-03-28T02:21:21.302 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_error_status_code", 2024-03-28T02:21:21.302 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.302 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.303 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.303 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "500", 2024-03-28T02:21:21.303 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "400", 2024-03-28T02:21:21.303 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "599", 2024-03-28T02:21:21.303 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.303 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.303 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.303 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.304 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.304 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.304 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.304 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_support", 2024-03-28T02:21:21.304 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.304 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.304 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.305 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.306 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.307 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.308 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "max_concurrent_snap_create": { 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_concurrent_snap_create", 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.309 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "mirror_snapshot_schedule": { 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.310 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "trash_purge_schedule": { 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "trash_purge_schedule", 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.311 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "restful", 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.312 INFO:teuthology.orchestra.run.smithi106.stdout: "enable_auth": { 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enable_auth", 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "key_file": { 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "key_file", 2024-03-28T02:21:21.313 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.314 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.315 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.316 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.317 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.318 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "server_addr": { 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_addr", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.319 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "server_port": { 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_port", 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.320 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rgw", 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.321 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.322 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.323 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.324 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.325 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rook", 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.326 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.327 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.328 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.329 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.330 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "storage_class": { 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "storage_class", 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "local", 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.331 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "selftest", 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.332 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.333 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.335 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "roption1": { 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "roption1", 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "roption2": { 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "roption2", 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "xyz", 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption1": { 2024-03-28T02:21:21.338 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption1", 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption2": { 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption2", 2024-03-28T02:21:21.339 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption3": { 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption3", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.340 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption4": { 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption4", 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.341 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption5": { 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption5", 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.342 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption6": { 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption6", 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.343 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption7": { 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption7", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "42", 2024-03-28T02:21:21.344 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "testkey": { 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "testkey", 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.345 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "testlkey": { 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "testlkey", 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.346 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "testnewline": { 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "testnewline", 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.347 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snap_schedule", 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_m_granularity": { 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "allow_m_granularity", 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.348 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "dump_on_update": { 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "dump_on_update", 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.349 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "dump database to debug log on update", 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.350 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.351 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.352 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.353 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.354 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "stats", 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.355 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.356 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.357 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.358 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.359 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "status", 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.360 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.361 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.362 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.363 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.364 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "telegraf", 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "address": { 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "address", 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.365 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "15", 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.366 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.367 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.368 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.369 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.370 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "telemetry", 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_basic": { 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_basic", 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T02:21:21.372 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_crash": { 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_crash", 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.373 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_device": { 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_device", 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.374 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_ident": { 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_ident", 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_perf": { 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_perf", 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "contact": { 2024-03-28T02:21:21.376 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "contact", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "description": { 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "description", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.377 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "device_url": { 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "device_url", 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.378 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "enabled": { 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enabled", 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.379 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "24", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "8", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.380 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "last_opt_revision": { 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "last_opt_revision", 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "1", 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "leaderboard": { 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "leaderboard", 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "leaderboard_description": { 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "leaderboard_description", 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.383 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.384 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.385 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.386 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.387 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "organization": { 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "organization", 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.388 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "proxy": { 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "proxy", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.389 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "url": { 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "url", 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.390 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "test_orchestrator", 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.391 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.392 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.394 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.395 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "volumes", 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.396 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.397 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.398 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.399 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "max_concurrent_clones": { 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_concurrent_clones", 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "4", 2024-03-28T02:21:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "periodic_async_work": { 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "periodic_async_work", 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Periodically check for async work", 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "snapshot_clone_delay": { 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snapshot_clone_delay", 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.403 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "snapshot_clone_no_wait": { 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.404 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix", 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.405 INFO:teuthology.orchestra.run.smithi106.stdout: "discovery_interval": { 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "discovery_interval", 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "100", 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.406 INFO:teuthology.orchestra.run.smithi106.stdout: "identifier": { 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "identifier", 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.407 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.408 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.409 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.410 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.411 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.412 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "zabbix_host": { 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix_host", 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.413 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "zabbix_port": { 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix_port", 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10051", 2024-03-28T02:21:21.414 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "zabbix_sender": { 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix_sender", 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.415 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: "modules": [ 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm", 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: "dashboard", 2024-03-28T02:21:21.416 INFO:teuthology.orchestra.run.smithi106.stdout: "iostat", 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "nfs", 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "restful", 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_schedule" 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "available_modules": [ 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "alerts", 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.417 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.418 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.419 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.420 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.421 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.422 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_destination": { 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_destination", 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.423 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Email address to send alerts to", 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_from_name": { 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_from_name", 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "Ceph", 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.424 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Email From: name", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_host": { 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_host", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SMTP server", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_password": { 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_password", 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Password to authenticate with", 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.426 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_port": { 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_port", 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "465", 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SMTP port", 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_sender": { 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_sender", 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SMTP envelope sender", 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.428 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_ssl": { 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_ssl", 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "smtp_user": { 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "smtp_user", 2024-03-28T02:21:21.429 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "User to authenticate as", 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.430 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "balancer", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "active": { 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "active", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.431 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "begin_time": { 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "begin_time", 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0000", 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.432 INFO:teuthology.orchestra.run.smithi106.stdout: "begin_weekday": { 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "begin_weekday", 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "0", 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "6", 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_compat_max_iterations": { 2024-03-28T02:21:21.433 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crush_compat_max_iterations", 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "25", 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "250", 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_compat_metrics": { 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crush_compat_metrics", 2024-03-28T02:21:21.434 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_compat_step": { 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crush_compat_step", 2024-03-28T02:21:21.435 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0.5", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "0.001", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "0.999", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "aggressiveness of optimization", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "end_time": { 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "end_time", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.436 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "2359", 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "end_weekday": { 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "end_weekday", 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.437 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "0", 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "6", 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.438 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.439 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.440 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.441 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.442 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "min_score": { 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "min_score", 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.443 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "mode": { 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mode", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "upmap", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-compat", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "none", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "read", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap", 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap-read" 2024-03-28T02:21:21.444 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Balancer mode", 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_ids": { 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pool_ids", 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T02:21:21.445 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.446 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap_max_deviation": { 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "upmap_max_deviation", 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.447 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "upmap_max_optimizations": { 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "upmap_max_optimizations", 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.448 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cephadm", 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "agent_down_multiplier": { 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "agent_down_multiplier", 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "3.0", 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.449 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "agent_refresh_rate": { 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "agent_refresh_rate", 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "20", 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.450 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "agent_starting_port": { 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "agent_starting_port", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "4721", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.451 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_ptrace": { 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "allow_ptrace", 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.452 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "autotune_interval": { 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "autotune_interval", 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "autotune_memory_target_ratio": { 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0.7", 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.454 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm_log_destination": { 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cephadm_log_destination", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "file", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "file,syslog", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "syslog" 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.455 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "cgroups_split": { 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cgroups_split", 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.456 INFO:teuthology.orchestra.run.smithi106.stdout: "config_checks_enabled": { 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "config_checks_enabled", 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "config_dashboard": { 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "config_dashboard", 2024-03-28T02:21:21.457 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_alertmanager": { 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_alertmanager", 2024-03-28T02:21:21.458 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_base": { 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_base", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.459 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Container image name, without the tag", 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_elasticsearch": { 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_elasticsearch", 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T02:21:21.460 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "elasticsearch container image", 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_grafana": { 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_grafana", 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.461 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_haproxy": { 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_haproxy", 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.462 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "HAproxy container image", 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_jaeger_agent": { 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_jaeger_agent", 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Jaeger agent container image", 2024-03-28T02:21:21.463 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_jaeger_collector": { 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_jaeger_collector", 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Jaeger collector container image", 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.464 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_jaeger_query": { 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_jaeger_query", 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Jaeger query container image", 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_keepalived": { 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_keepalived", 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Keepalived container image", 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.466 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_loki": { 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_loki", 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Loki container image", 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_node_exporter": { 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_node_exporter", 2024-03-28T02:21:21.467 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_nvmeof": { 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_nvmeof", 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.468 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Nvme-of container image", 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_prometheus": { 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_prometheus", 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.469 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Prometheus container image", 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_promtail": { 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_promtail", 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T02:21:21.470 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Promtail container image", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image_snmp_gateway": { 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_image_snmp_gateway", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.471 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "SNMP Gateway container image", 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "container_init": { 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "container_init", 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T02:21:21.472 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "daemon_cache_timeout": { 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "daemon_cache_timeout", 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T02:21:21.473 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "default_cephadm_command_timeout": { 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "900", 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.474 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "default_registry": { 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "default_registry", 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "docker.io", 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.475 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "device_cache_timeout": { 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "device_cache_timeout", 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "1800", 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "seconds to cache device inventory", 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "device_enhanced_scan": { 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "device_enhanced_scan", 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "facts_cache_timeout": { 2024-03-28T02:21:21.477 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "facts_cache_timeout", 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "seconds to cache host facts data", 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "host_check_interval": { 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "host_check_interval", 2024-03-28T02:21:21.478 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to perform a host check", 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "hw_monitoring": { 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "hw_monitoring", 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.479 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "inventory_list_all": { 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "inventory_list_all", 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.480 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.481 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "log_refresh_metadata": { 2024-03-28T02:21:21.482 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_refresh_metadata", 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.483 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.484 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T02:21:21.486 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T02:21:21.487 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "*", 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "max_count_per_host": { 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_count_per_host", 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.488 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "max number of daemons per service per host", 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd_draining_count": { 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_osd_draining_count", 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.489 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "migration_current": { 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "migration_current", 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.490 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "internal - do not modify", 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "mode": { 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mode", 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "root", 2024-03-28T02:21:21.491 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm-package", 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "root" 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "oob_default_addr": { 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "oob_default_addr", 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.492 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "169.254.1.1", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "prometheus_alerts_path": { 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "prometheus_alerts_path", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.493 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_insecure": { 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_insecure", 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.494 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_password": { 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_password", 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.495 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_url": { 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_url", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.496 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "registry_username": { 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "registry_username", 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "secure_monitoring_stack": { 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "secure_monitoring_stack", 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.498 INFO:teuthology.orchestra.run.smithi106.stdout: "service_discovery_port": { 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "service_discovery_port", 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8765", 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "cephadm service discovery port", 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "ssh_config_file": { 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssh_config_file", 2024-03-28T02:21:21.499 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "use_agent": { 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "use_agent", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.500 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "use_repo_digest": { 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "use_repo_digest", 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.501 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_on_failed_host_check": { 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_on_failed_host_check", 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.502 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_on_stray_daemons": { 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_on_stray_daemons", 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.503 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_on_stray_hosts": { 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_on_stray_hosts", 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.504 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crash", 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.505 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.506 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.509 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.509 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.509 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.509 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.509 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.509 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.509 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.511 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.512 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "retain_interval": { 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "retain_interval", 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "31536000", 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T02:21:21.513 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_recent_interval": { 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_recent_interval", 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "1209600", 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T02:21:21.514 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "dashboard", 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.515 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.516 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.517 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "AUDIT_API_ENABLED": { 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.518 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.519 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.520 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T02:21:21.521 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.522 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.523 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.524 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.525 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.526 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.527 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "admin", 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.528 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_URL": { 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_URL", 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.530 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "admin", 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T02:21:21.531 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.532 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.533 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.534 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "{}", 2024-03-28T02:21:21.535 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.536 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.537 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.538 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.540 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T02:21:21.541 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T02:21:21.542 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T02:21:21.543 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.544 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T02:21:21.545 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.546 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8", 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.547 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "45", 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.548 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.549 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "admin", 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T02:21:21.551 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.553 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.554 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:21.555 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "cross_origin_url": { 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cross_origin_url", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.556 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "crt_file": { 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "crt_file", 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.557 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "debug": { 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "debug", 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Enable/disable debug options", 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.558 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "jwt_token_ttl": { 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "jwt_token_ttl", 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "28800", 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.559 INFO:teuthology.orchestra.run.smithi106.stdout: "key_file": { 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "key_file", 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.560 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.561 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.562 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.563 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.564 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "motd": { 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "motd", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "The message of the day", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.565 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "redirect_resolve_ip_addr": { 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.566 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "server_addr": { 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_addr", 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "::", 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.567 INFO:teuthology.orchestra.run.smithi106.stdout: "server_port": { 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_port", 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8080", 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "ssl": { 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssl", 2024-03-28T02:21:21.568 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "ssl_server_port": { 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssl_server_port", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.569 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8443", 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_behaviour": { 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_behaviour", 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.570 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "redirect", 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "redirect" 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_error_status_code": { 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_error_status_code", 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.571 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "500", 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "400", 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "599", 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "url_prefix": { 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "url_prefix", 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.572 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "devicehealth", 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.573 INFO:teuthology.orchestra.run.smithi106.stdout: "enable_monitoring": { 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enable_monitoring", 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "monitor device health metrics", 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.574 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.575 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.576 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.577 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.578 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "mark_out_threshold": { 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mark_out_threshold", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "2419200", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.579 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": { 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pool_name", 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "device_health_metrics", 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T02:21:21.580 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "retention_period": { 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "retention_period", 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "15552000", 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how long to retain device health metrics", 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.581 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "scrape_frequency": { 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "scrape_frequency", 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "86400", 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.582 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "self_heal": { 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "self_heal", 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.583 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "warn_threshold": { 2024-03-28T02:21:21.584 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "warn_threshold", 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "7257600", 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.585 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "diskprediction_local", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.586 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.587 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.588 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.589 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "predict_interval": { 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "predict_interval", 2024-03-28T02:21:21.590 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "86400", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "predictor_model": { 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "predictor_model", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.591 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "prophetstor", 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.592 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "600", 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.593 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "influx", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": false, 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "influxdb python module not found", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "batch_size": { 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "batch_size", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5000", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.594 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "database": { 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "database", 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "ceph", 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.595 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "hostname": { 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "hostname", 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "InfluxDB server hostname", 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.596 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "30", 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "5", 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.597 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.598 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.599 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.600 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.601 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "password": { 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "password", 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.602 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "password of InfluxDB server user", 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "port": { 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "port", 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.603 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "8086", 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "InfluxDB server port", 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "ssl": { 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ssl", 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.604 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "false", 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "threads": { 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "threads", 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.605 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "32", 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "username": { 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "username", 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.606 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "username of InfluxDB server user", 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "verify_ssl": { 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "verify_ssl", 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.607 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "true", 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "insights", 2024-03-28T02:21:21.608 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.609 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.610 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.611 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.612 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.613 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "iostat", 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.614 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.615 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.616 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.617 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.618 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.619 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "k8sevents", 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_event_retention_days": { 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "ceph_event_retention_days", 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "7", 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.620 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "config_check_secs": { 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "config_check_secs", 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "10", 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.621 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.622 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.623 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.624 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.625 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.626 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "localpool", 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "failure_domain": { 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "failure_domain", 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.627 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "host", 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "failure domain for any created local pool", 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.628 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.631 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.632 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": { 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "min_size", 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "default min_size for any created local pool", 2024-03-28T02:21:21.633 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "num_rep": { 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "num_rep", 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "3", 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "default replica count for any created local pool", 2024-03-28T02:21:21.634 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": { 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pg_num", 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "128", 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.635 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "prefix": { 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "prefix", 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "name prefix for any created local pool", 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.636 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "subtree": { 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "subtree", 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "rack", 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.637 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mds_autoscaler", 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.638 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.639 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.640 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.641 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.642 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mirroring", 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.643 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.645 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.646 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.647 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "nfs", 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.648 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.649 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.650 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.651 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.652 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "orchestrator", 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "fail_fs": { 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "fail_fs", 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.653 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.654 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.655 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.656 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.657 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.658 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "orchestrator": { 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "orchestrator", 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm", 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "rook", 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "test_orchestrator" 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.659 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Orchestrator backend", 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "osd_perf_query", 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.660 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.661 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.662 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.663 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.664 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.665 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "osd_support", 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.666 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.667 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.668 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.669 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.670 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "pg_autoscaler", 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.671 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.674 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.675 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.676 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "threshold": { 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "threshold", 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "3.0", 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1.0", 2024-03-28T02:21:21.677 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "scaling threshold", 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "progress", 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.678 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_pg_recovery_event": { 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "allow_pg_recovery_event", 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "enabled": { 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enabled", 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.680 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.681 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.682 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.683 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.684 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "max_completed_events": { 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_completed_events", 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.685 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "50", 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "number of past completed events to remember", 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "sleep_interval": { 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "sleep_interval", 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "5", 2024-03-28T02:21:21.686 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "how long the module is going to sleep", 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "prometheus", 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.687 INFO:teuthology.orchestra.run.smithi106.stdout: "cache": { 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "cache", 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "exclude_perf_counters": { 2024-03-28T02:21:21.688 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "exclude_perf_counters", 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.689 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.690 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.691 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.692 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.693 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_stats_pools": { 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_stats_pools", 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.694 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.695 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "300", 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "scrape_interval": { 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "scrape_interval", 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "15.0", 2024-03-28T02:21:21.696 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "server_addr": { 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_addr", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "::", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.697 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "server_port": { 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_port", 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "9283", 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.698 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "stale_cache_strategy": { 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "stale_cache_strategy", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "log", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.699 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_behaviour": { 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_behaviour", 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "default", 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "default", 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "error" 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.700 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "standby_error_status_code": { 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "standby_error_status_code", 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "500", 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "400", 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "599", 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.701 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_support", 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.702 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.703 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.704 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.705 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.706 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "max_concurrent_snap_create": { 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_concurrent_snap_create", 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10", 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "mirror_snapshot_schedule": { 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "trash_purge_schedule": { 2024-03-28T02:21:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "trash_purge_schedule", 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.710 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "restful", 2024-03-28T02:21:21.710 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.710 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.710 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.710 INFO:teuthology.orchestra.run.smithi106.stdout: "enable_auth": { 2024-03-28T02:21:21.710 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enable_auth", 2024-03-28T02:21:21.710 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.710 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.710 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.713 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.713 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.713 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.713 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.713 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.713 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "key_file": { 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "key_file", 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.714 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.715 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.716 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.717 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.718 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "server_addr": { 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_addr", 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.719 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "server_port": { 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "server_port", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.720 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rgw", 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.721 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.722 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.723 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.724 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.725 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rook", 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.726 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.727 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.728 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.729 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.730 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "storage_class": { 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "storage_class", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "local", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.731 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "selftest", 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.732 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.733 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.735 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.736 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "roption1": { 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "roption1", 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.737 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "roption2": { 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "roption2", 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "xyz", 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.738 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption1": { 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption1", 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.739 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption2": { 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption2", 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption3": { 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption3", 2024-03-28T02:21:21.740 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "float", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption4": { 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption4", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.741 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption5": { 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption5", 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.742 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption6": { 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption6", 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.743 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "rwoption7": { 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rwoption7", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "1", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "42", 2024-03-28T02:21:21.744 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "testkey": { 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "testkey", 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.745 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "testlkey": { 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "testlkey", 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.746 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "testnewline": { 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "testnewline", 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.747 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snap_schedule", 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_m_granularity": { 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "allow_m_granularity", 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.748 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "dump_on_update": { 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "dump_on_update", 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.749 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "dump database to debug log on update", 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.750 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.751 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.752 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "stats", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.755 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.756 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.757 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.758 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.759 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "status", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.760 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.761 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.763 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "telegraf", 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.764 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "address": { 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "address", 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.765 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "15", 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.766 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.767 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.768 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.769 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.770 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "telemetry", 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_basic": { 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_basic", 2024-03-28T02:21:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_crash": { 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_crash", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.772 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_device": { 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_device", 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.773 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_ident": { 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_ident", 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.774 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "channel_perf": { 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "channel_perf", 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.775 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "contact": { 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "contact", 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.776 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "description": { 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "description", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.777 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "device_url": { 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "device_url", 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "enabled": { 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "enabled", 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.779 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "24", 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "8", 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "last_opt_revision": { 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "last_opt_revision", 2024-03-28T02:21:21.780 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "1", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "leaderboard": { 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "leaderboard", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.781 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "leaderboard_description": { 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "leaderboard_description", 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.782 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.783 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.784 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.785 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.786 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "organization": { 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "organization", 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "proxy": { 2024-03-28T02:21:21.788 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "proxy", 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "url": { 2024-03-28T02:21:21.789 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "url", 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "test_orchestrator", 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.791 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.792 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.793 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.794 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "volumes", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.796 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.797 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.798 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.799 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: "max_concurrent_clones": { 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "max_concurrent_clones", 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "4", 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "periodic_async_work": { 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "periodic_async_work", 2024-03-28T02:21:21.801 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Periodically check for async work", 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "snapshot_clone_delay": { 2024-03-28T02:21:21.802 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snapshot_clone_delay", 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "0", 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "snapshot_clone_no_wait": { 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T02:21:21.803 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "True", 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix", 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "can_run": true, 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "error_string": "", 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "module_options": { 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "discovery_interval": { 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "discovery_interval", 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "uint", 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "100", 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.805 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "identifier": { 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "identifier", 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.806 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "interval": { 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "interval", 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "secs", 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "60", 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.807 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "log_level": { 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_level", 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.808 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster": { 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster", 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.809 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_cluster_level": { 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_cluster_level", 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.810 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "info", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [ 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "critical", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "debug", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "error", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "info", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "warning" 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.811 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": { 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "log_to_file", 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "bool", 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "False", 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.812 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "zabbix_host": { 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix_host", 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "", 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.813 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "zabbix_port": { 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix_port", 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "int", 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "10051", 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "zabbix_sender": { 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "zabbix_sender", 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "str", 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "level": "advanced", 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 0, 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "min": "", 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "max": "", 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "enum_allowed": [], 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "desc": "", 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "long_desc": "", 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "tags": [], 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: "see_also": [] 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "services": { 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "dashboard": "https://172.21.15.106:8443/" 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "always_on_modules": { 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "octopus": [ 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "balancer", 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "crash", 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "devicehealth", 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "orchestrator", 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscaler", 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "progress", 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_support", 2024-03-28T02:21:21.816 INFO:teuthology.orchestra.run.smithi106.stdout: "status", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "telemetry", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "volumes" 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "pacific": [ 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "balancer", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "crash", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "devicehealth", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "orchestrator", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscaler", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "progress", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_support", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "status", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "telemetry", 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "volumes" 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.817 INFO:teuthology.orchestra.run.smithi106.stdout: "quincy": [ 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "balancer", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "crash", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "devicehealth", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "orchestrator", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscaler", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "progress", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_support", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "status", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "telemetry", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "volumes" 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "reef": [ 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "balancer", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "crash", 2024-03-28T02:21:21.818 INFO:teuthology.orchestra.run.smithi106.stdout: "devicehealth", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "orchestrator", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscaler", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "progress", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_support", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "status", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "telemetry", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "volumes" 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "squid": [ 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "balancer", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "crash", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "devicehealth", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "orchestrator", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscaler", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "progress", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_support", 2024-03-28T02:21:21.819 INFO:teuthology.orchestra.run.smithi106.stdout: "status", 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: "telemetry", 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: "volumes" 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: "last_failure_osd_epoch": 94, 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: "active_clients": [ 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "devicehealth", 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:0", 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3098117168 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.820 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "libcephsqlite", 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:0", 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1550220334 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "rbd_support", 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-03-28T02:21:21.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:0", 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2802515328 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snap_schedule", 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:0", 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2239395332 2024-03-28T02:21:21.822 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "snap_schedule", 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:0", 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3752545971 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "volumes", 2024-03-28T02:21:21.823 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-03-28T02:21:21.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-03-28T02:21:21.824 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-03-28T02:21:21.824 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:0", 2024-03-28T02:21:21.824 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3172383003 2024-03-28T02:21:21.824 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.824 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.824 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:21.824 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-28T02:21:21.824 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:21:21.825 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:20 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3639808703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:21:21.825 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:20 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3679040729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:21:21.826 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:20 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2048864577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:21:21.826 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:20 smithi178 ceph-mon[36521]: pgmap v313: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:21:21.826 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:20 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/976979716' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T02:21:21.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:20 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3639808703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:21:21.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:20 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3679040729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:21:21.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:20 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2048864577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:21:21.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:20 smithi153 ceph-mon[36228]: pgmap v313: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:21:21.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:20 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/976979716' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T02:21:21.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbad00a2b10 msgr2=0x7fbad00a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:21.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbad00a2b10 0x7fbad00a4fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009b50 tx=0x7fbad4005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:21.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101ff0 msgr2=0x7fbae41984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:21.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101ff0 0x7fbae41984b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009230 tx=0x7fbacc02cb30 comp rx=0 tx=0).stop 2024-03-28T02:21:21.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 shutdown_connections 2024-03-28T02:21:21.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fbad00a2b10 0x7fbad00a4fd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:21.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbae4107640 0x7fbae419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:21.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbae4101ff0 0x7fbae41984b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:21.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 --2- 172.21.15.106:0/976979716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae4101630 0x7fbae4197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:21.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 >> 172.21.15.106:0/976979716 conn(0x7fbae40fd140 msgr2=0x7fbae40fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:21.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 shutdown_connections 2024-03-28T02:21:21.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:20.639+0000 7fbaea481640 1 -- 172.21.15.106:0/976979716 wait complete. 2024-03-28T02:21:21.833 DEBUG:tasks.check_counter:Getting stats from a 2024-03-28T02:21:21.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-28T02:21:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:20 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3639808703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:21:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:20 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3679040729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:21:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:20 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2048864577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:21:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:20 smithi106 ceph-mon[31965]: pgmap v313: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T02:21:21.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:20 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/976979716' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T02:21:22.085 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:21:22.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:22 smithi106 ceph-mon[31965]: pgmap v314: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-28T02:21:22.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:22 smithi153 ceph-mon[36228]: pgmap v314: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-28T02:21:23.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:22 smithi178 ceph-mon[36521]: pgmap v314: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-28T02:21:23.909 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:21:23.909 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:23.909 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 1468, 2024-03-28T02:21:23.909 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 1470, 2024-03-28T02:21:23.909 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 439875, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 466825, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 6, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 2, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.133335017, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.046547941, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.033700130, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.040719396, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1468, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.051846464, 2024-03-28T02:21:23.910 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000035317 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 239, 2024-03-28T02:21:23.911 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 228, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 1470046, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 127838, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 4, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 3, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.043636205, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.015335818, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.032373097, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.004623529, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 227, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.015478824, 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000068188 2024-03-28T02:21:23.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 1468656, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 126816 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 613, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 766, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 152688, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 626530, 2024-03-28T02:21:23.913 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 7, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 5, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.071394031, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.024772224, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.016500739, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.023110250, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 764, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.032808028, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000042942 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.914 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 433504 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "cct": { 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "total_workers": 1, 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "unhealthy_workers": 0 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-MDSRank": { 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 3380, 2024-03-28T02:21:23.915 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.093046163, 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000027528 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "mds": { 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "request": 0, 2024-03-28T02:21:23.916 INFO:teuthology.orchestra.run.smithi106.stdout: "reply": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "reply_latency": { 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_reply": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "forward": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_commit": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_split": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_merge": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes": 0, 2024-03-28T02:21:23.917 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_top": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_bottom": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pinned": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_expired": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_with_caps": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "caps": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "subtrees": 1, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_hit": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_forward": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_discover": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:23.918 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_lock": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "load_cent": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "q": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "exported": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "exported_inodes": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "imported": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "imported_inodes": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rfiles": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rbytes": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rsnaps": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:23.919 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_grant": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps": 0, 2024-03-28T02:21:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps_dirty": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_cap_release": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "process_request_cap_release": 0 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache": { 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update_receipt": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_try_discover": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_send_discover": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_handle_discover": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_created": 0, 2024-03-28T02:21:23.921 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_enqueued": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_migrated": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_started": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_completed": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_flush": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_inodestats": 0 2024-03-28T02:21:23.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log": { 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "evlrg": 0, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "evadd": 0, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "evex": 0, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "evtrm": 0, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "ev": 4, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "evexg": 0, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "evexd": 0, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "segadd": 0, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "segex": 0, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "segtrm": 0, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "seg": 2, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "segmjr": 2, 2024-03-28T02:21:23.923 INFO:teuthology.orchestra.run.smithi106.stdout: "segexg": 0, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "segexd": 0, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "expos": 4194304, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "wrpos": 4195424, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "rdpos": 4195283, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "jlat": { 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "replayed": 4 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_mem": { 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "ino": 3, 2024-03-28T02:21:23.924 INFO:teuthology.orchestra.run.smithi106.stdout: "ino+": 3, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "ino-": 0, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "dir": 1, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "dir+": 1, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "dir-": 0, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "dn": 0, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "dn+": 0, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "dn-": 0, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "cap": 0, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "cap+": 0, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "cap-": 0, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "rss": 52804, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "heap": 215420 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_server": { 2024-03-28T02:21:23.925 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_client_request": 0, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_request": 0, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_session": 0, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_peer_request": 0, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "req_create_latency": { 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getattr_latency": { 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "req_link_latency": { 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookup_latency": { 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.927 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupino_latency": { 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupname_latency": { 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.928 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lssnap_latency": { 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.929 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.930 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mkdir_latency": { 2024-03-28T02:21:23.930 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.930 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.930 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.930 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mknod_latency": { 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mksnap_latency": { 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "req_open_latency": { 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "req_readdir_latency": { 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rename_latency": { 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmdir_latency": { 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setattr_latency": { 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setlayout_latency": { 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setxattr_latency": { 2024-03-28T02:21:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "req_symlink_latency": { 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "req_unlink_latency": { 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_sessions": { 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "session_count": 0, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "session_add": 1, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "session_remove": 1, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_open": 1, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_stale": 0, 2024-03-28T02:21:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "total_load": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "average_load": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "avg_session_uptime": 1350, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "mempool": { 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:23.938 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:23.939 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_items": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:23.940 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_bytes": 197553, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_items": 57, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bytes": 0, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_items": 0, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:23.941 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_items": 183, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_items": 0, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_bytes": 8320, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_items": 13, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_items": 0, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_items": 0 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter": { 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "op_active": 0, 2024-03-28T02:21:23.942 INFO:teuthology.orchestra.run.smithi106.stdout: "op_laggy": 0, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send": 2078, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send_bytes": 21, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "op_resend": 3, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "op_reply": 2078, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "op_latency": { 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 2078, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 1.056973692, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000508649 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "op_inflight": 0, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "oplen_avg": { 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 2078, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 2079 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.943 INFO:teuthology.orchestra.run.smithi106.stdout: "op": 2078, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "op_r": 2078, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "op_w": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "op_rmw": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "op_pg": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_stat": 1338, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_create": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_read": 738, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_write": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writefull": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writesame": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_append": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_zero": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_truncate": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_delete": 0, 2024-03-28T02:21:23.944 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_mapext": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_getxattr": 1, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_setxattr": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_call": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_watch": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_notify": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_other": 0, 2024-03-28T02:21:23.945 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_active": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_send": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_resend": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_ping": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_active": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_send": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_resend": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_active": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_send": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_resend": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_active": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_send": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_resend": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "command_active": 0, 2024-03-28T02:21:23.946 INFO:teuthology.orchestra.run.smithi106.stdout: "command_send": 0, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "command_resend": 0, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "map_epoch": 102, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "map_full": 0, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "map_inc": 13, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_sessions": 6, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_open": 6, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_close": 0, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_laggy": 0, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_wr": 0, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_rd": 2, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_del": 0 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "oft": { 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_objs": 0, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:23.947 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_updates": 0, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_removes": 0 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "osd-slow-ops": { 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_ops_count": 0 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "purge_queue": { 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing": 0, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_high_water": 0, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed_ops": 0, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed": 0, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:23.948 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 2316, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 1839072, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 2316, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 2316, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 1839072, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:23.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 2078, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 1125, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 2078, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 1125, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.950 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 1024, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 2078, 2024-03-28T02:21:23.951 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 2078, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 2078, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 2078, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:23.952 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 0, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 0, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle-0x564895f946e0": { 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:23.953 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 0, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 0, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:23.954 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:21:24.631 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-28T02:21:24.631 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-28T02:21:24.631 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-03-28T02:21:24.631 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-03-28T02:21:24.631 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-03-28T02:21:24.631 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-28T02:21:24.631 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T02:21:24.631 DEBUG:tasks.check_counter:Getting stats from d 2024-03-28T02:21:24.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-28T02:21:24.880 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:21:24.907 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:24 smithi106 ceph-mon[31965]: pgmap v315: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 91 B/s wr, 11 op/s 2024-03-28T02:21:24.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:24 smithi153 ceph-mon[36228]: pgmap v315: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 91 B/s wr, 11 op/s 2024-03-28T02:21:25.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:24 smithi178 ceph-mon[36521]: pgmap v315: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 91 B/s wr, 11 op/s 2024-03-28T02:21:26.701 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:21:26.701 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:26.701 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 4, 2024-03-28T02:21:26.701 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 4, 2024-03-28T02:21:26.701 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 3024, 2024-03-28T02:21:26.701 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 448, 2024-03-28T02:21:26.701 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 4, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 0, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.001307039, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.000037388, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.000447661, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 3, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000687680, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000229226 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:26.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 2, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 161, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 320, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 183552, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 5, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 3, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.014999446, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.010498541, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.002106063, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T02:21:26.703 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 158, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.020748975, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000131322 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 183552 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 222, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 212, 2024-03-28T02:21:26.704 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 1437664, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 126816, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 2, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 1, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.042437256, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.014313314, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.031488424, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.004473658, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 211, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.011405461, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000054054 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:26.705 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 1437664, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 126816 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "cct": { 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "total_workers": 0, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "unhealthy_workers": 0 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "mempool": { 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:26.706 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:26.707 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_items": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:26.708 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_bytes": 148955, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_items": 30, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bytes": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_items": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_bytes": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_items": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:26.709 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_items": 0, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_bytes": 928, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_items": 2, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_items": 0, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_items": 0 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 224, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 1406391, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:26.710 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 224, 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 224, 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 1406391, 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:26.711 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:21:26.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:26 smithi153 ceph-mon[36228]: pgmap v316: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T02:21:27.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:26 smithi106 ceph-mon[31965]: pgmap v316: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T02:21:27.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:26 smithi178 ceph-mon[36521]: pgmap v316: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T02:21:27.451 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-28T02:21:27.451 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-28T02:21:27.451 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-28T02:21:27.451 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-28T02:21:27.451 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-28T02:21:27.451 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-28T02:21:27.451 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-28T02:21:27.451 DEBUG:tasks.check_counter:Getting stats from g 2024-03-28T02:21:27.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-28T02:21:27.693 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:21:28.831 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:28 smithi106 ceph-mon[31965]: pgmap v317: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-03-28T02:21:28.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:28 smithi153 ceph-mon[36228]: pgmap v317: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-03-28T02:21:29.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:28 smithi178 ceph-mon[36521]: pgmap v317: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-03-28T02:21:29.411 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 738, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 740, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 247913, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 234685, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 6, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 2, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.084458822, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.035952330, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.017335022, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.022329469, 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:29.412 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 738, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.042816238, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000058016 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 1591, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 1580, 2024-03-28T02:21:29.413 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 1815405, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 558375, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 4, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 3, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.128588053, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.039163037, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.053403301, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.037938301, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1579, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.053152842, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000033662 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.414 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 1469200, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 128064 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 9, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 164, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 4692, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 438021, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 7, 2024-03-28T02:21:29.415 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 5, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.013863806, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.008957189, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.002780621, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.000055805, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 162, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.016445858, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000101517 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 437056 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "cct": { 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "total_workers": 1, 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "unhealthy_workers": 0 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-MDSRank": { 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 3552, 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.087013400, 2024-03-28T02:21:29.417 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000024497 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "mds": { 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "request": 0, 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "reply": 0, 2024-03-28T02:21:29.418 INFO:teuthology.orchestra.run.smithi106.stdout: "reply_latency": { 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_reply": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "forward": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_commit": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_split": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_merge": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_top": 0, 2024-03-28T02:21:29.419 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_bottom": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pinned": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_expired": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_with_caps": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "caps": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "subtrees": 1, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_hit": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_forward": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_discover": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_lock": 0, 2024-03-28T02:21:29.420 INFO:teuthology.orchestra.run.smithi106.stdout: "load_cent": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "q": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "exported": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "exported_inodes": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "imported": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "imported_inodes": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rfiles": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rbytes": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rsnaps": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:29.421 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_grant": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps_dirty": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_cap_release": 0, 2024-03-28T02:21:29.422 INFO:teuthology.orchestra.run.smithi106.stdout: "process_request_cap_release": 0 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache": { 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update_receipt": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_try_discover": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_send_discover": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_handle_discover": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_created": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_enqueued": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:29.423 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_migrated": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_started": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_completed": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_flush": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_inodestats": 0 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log": { 2024-03-28T02:21:29.424 INFO:teuthology.orchestra.run.smithi106.stdout: "evlrg": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "evadd": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "evex": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "evtrm": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "ev": 5, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "evexg": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "evexd": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "segadd": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "segex": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "segtrm": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "seg": 2, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "segmjr": 2, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "segexg": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "segexd": 0, 2024-03-28T02:21:29.425 INFO:teuthology.orchestra.run.smithi106.stdout: "expos": 4194304, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "wrpos": 4195815, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "rdpos": 4195674, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "jlat": { 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "replayed": 5 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_mem": { 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "ino": 3, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "ino+": 3, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "ino-": 0, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "dir": 1, 2024-03-28T02:21:29.426 INFO:teuthology.orchestra.run.smithi106.stdout: "dir+": 1, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "dir-": 0, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "dn": 0, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "dn+": 0, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "dn-": 0, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "cap": 0, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "cap+": 0, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "cap-": 0, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "rss": 49840, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "heap": 215420 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_server": { 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_client_request": 0, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:29.427 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_request": 0, 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_session": 0, 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_peer_request": 0, 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "req_create_latency": { 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getattr_latency": { 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:29.428 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "req_link_latency": { 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookup_latency": { 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:29.429 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupino_latency": { 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupname_latency": { 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lssnap_latency": { 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.432 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mkdir_latency": { 2024-03-28T02:21:29.432 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.432 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.432 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.432 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mknod_latency": { 2024-03-28T02:21:29.432 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mksnap_latency": { 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "req_open_latency": { 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.433 INFO:teuthology.orchestra.run.smithi106.stdout: "req_readdir_latency": { 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rename_latency": { 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.434 INFO:teuthology.orchestra.run.smithi106.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmdir_latency": { 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.435 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setattr_latency": { 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setlayout_latency": { 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setxattr_latency": { 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "req_symlink_latency": { 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "req_unlink_latency": { 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.438 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_sessions": { 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "session_count": 1, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "session_add": 2, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "session_remove": 1, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_open": 2, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_stale": 0, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "total_load": 0, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "average_load": 0, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "avg_session_uptime": 3134, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.439 INFO:teuthology.orchestra.run.smithi106.stdout: "mempool": { 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:29.440 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:29.441 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_items": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_bytes": 197473, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_items": 55, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bytes": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_items": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:29.442 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_items": 183, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_items": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_bytes": 8320, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_items": 13, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_items": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_items": 0 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.443 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter": { 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "op_active": 0, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "op_laggy": 0, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send": 2094, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send_bytes": 21, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "op_resend": 3, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "op_reply": 2094, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "op_latency": { 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 2094, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 1.732956716, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000827582 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "op_inflight": 0, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "oplen_avg": { 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 2094, 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 2095 2024-03-28T02:21:29.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "op": 2094, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "op_r": 2094, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "op_w": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "op_rmw": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "op_pg": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_stat": 1348, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_create": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_read": 744, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_write": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writefull": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writesame": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_append": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_zero": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_truncate": 0, 2024-03-28T02:21:29.445 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_delete": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_mapext": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_getxattr": 1, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_setxattr": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_call": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_watch": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_notify": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:29.446 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_other": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_active": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_send": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_resend": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_ping": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_active": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_send": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_resend": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_active": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_send": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_resend": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_active": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_send": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_resend": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "command_active": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "command_send": 0, 2024-03-28T02:21:29.447 INFO:teuthology.orchestra.run.smithi106.stdout: "command_resend": 0, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "map_epoch": 102, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "map_full": 0, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "map_inc": 13, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_sessions": 6, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_open": 6, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_close": 0, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_laggy": 0, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_wr": 0, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_rd": 2, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_del": 0 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "oft": { 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_objs": 0, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:29.448 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_updates": 0, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_removes": 0 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "osd-slow-ops": { 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_ops_count": 0 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "purge_queue": { 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing": 0, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_high_water": 0, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed_ops": 0, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed": 0, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 2334, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 1842749, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 2334, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 2334, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 1842749, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.450 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 2094, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 1516, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 2094, 2024-03-28T02:21:29.451 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 1516, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 1024, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 2094, 2024-03-28T02:21:29.452 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 2094, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 2094, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 2094, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-28T02:21:29.453 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 0, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 0, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle-0x5576a1ebe6e0": { 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:29.454 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 0, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 0, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:29.455 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:21:30.212 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-28T02:21:30.212 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-28T02:21:30.212 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-03-28T02:21:30.212 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-03-28T02:21:30.212 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-03-28T02:21:30.212 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-03-28T02:21:30.212 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T02:21:30.213 DEBUG:tasks.check_counter:Getting stats from j 2024-03-28T02:21:30.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-28T02:21:30.455 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T02:21:30.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:30 smithi106 ceph-mon[31965]: pgmap v318: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-28T02:21:30.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:30 smithi153 ceph-mon[36228]: pgmap v318: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-28T02:21:31.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:30 smithi178 ceph-mon[36521]: pgmap v318: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-28T02:21:31.878 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:31 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:21:31.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:31 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:21:32.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:31 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T02:21:32.293 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-28T02:21:32.293 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:32.293 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 4, 2024-03-28T02:21:32.293 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 4, 2024-03-28T02:21:32.293 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 3024, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 448, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 4, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 0, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.001301091, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.000035061, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.000510180, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 3, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000663737, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000221245 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:32.294 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 2, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 163, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 320, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 183904, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 5, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 3, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.013662378, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.009439638, 2024-03-28T02:21:32.295 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.001812343, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 160, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.017346346, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000108414 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 183904 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.296 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 224, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 214, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 1438208, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 128064, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 2, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 1, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.043101598, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.015169665, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.031494202, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.004572045, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 213, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.013251816, 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000062215 2024-03-28T02:21:32.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 1438208, 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 128064 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "cct": { 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "total_workers": 0, 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "unhealthy_workers": 0 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "mempool": { 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:32.298 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:32.299 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_items": 0, 2024-03-28T02:21:32.300 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_bytes": 148714, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_items": 30, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bytes": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_items": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_bytes": 0, 2024-03-28T02:21:32.301 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_items": 0, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_items": 0, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_bytes": 928, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_items": 2, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_items": 0, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_items": 0 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-28T02:21:32.302 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 226, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 1406651, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 226, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 226, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 1406651, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-28T02:21:32.303 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-28T02:21:33.085 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-28T02:21:33.085 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-28T02:21:33.085 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-28T02:21:33.086 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-28T02:21:33.086 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-28T02:21:33.086 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-28T02:21:33.086 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-28T02:21:33.086 DEBUG:tasks.check_counter:Getting stats from b 2024-03-28T02:21:33.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-28T02:21:33.339 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: pgmap v319: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:33 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: pgmap v319: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:33 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: pgmap v319: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:33.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:33 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:34.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:34 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-03-28T02:21:34.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:34 smithi153 ceph-mon[36228]: Unable to set osd_memory_target on smithi153 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-28T02:21:34.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:34 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi178 to 174.1M 2024-03-28T02:21:34.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:34 smithi153 ceph-mon[36228]: Unable to set osd_memory_target on smithi178 to 182604083: error parsing value: Value '182604083' is below minimum 939524096 2024-03-28T02:21:34.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:34 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:34.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:34 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:34.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:34 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-03-28T02:21:34.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:34 smithi106 ceph-mon[31965]: Unable to set osd_memory_target on smithi153 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-28T02:21:34.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:34 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi178 to 174.1M 2024-03-28T02:21:34.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:34 smithi106 ceph-mon[31965]: Unable to set osd_memory_target on smithi178 to 182604083: error parsing value: Value '182604083' is below minimum 939524096 2024-03-28T02:21:34.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:34 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:34.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:34 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:34.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:34 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-03-28T02:21:34.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:34 smithi178 ceph-mon[36521]: Unable to set osd_memory_target on smithi153 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-28T02:21:34.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:34 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi178 to 174.1M 2024-03-28T02:21:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:34 smithi178 ceph-mon[36521]: Unable to set osd_memory_target on smithi178 to 182604083: error parsing value: Value '182604083' is below minimum 939524096 2024-03-28T02:21:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:34 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:34.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:34 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:35.240 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-28T02:21:35.240 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:35.240 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 4465, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 4200, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 2735618, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 5870179, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 12, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.364985819, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.107646086, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.116440696, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.110899121, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4202, 2024-03-28T02:21:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.514443549, 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000122428 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1546, 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000242771, 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000157 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1516704, 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 445408 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 8991, 2024-03-28T02:21:35.242 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 9423, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 2851176, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 6825913, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 25, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 7, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.694248448, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.228889323, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.454479158, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.133480004, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 9417, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2.017259908, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000214214 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.243 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 9690, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.003921805, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000404 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 400816, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 261952 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 4509, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 4049, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1312244, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 5771320, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 30, 2024-03-28T02:21:35.244 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 8, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.395620897, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.132734675, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.096885030, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.127564034, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4048, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.533458017, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000131783 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1750, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000265796, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000151 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.245 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 45248, 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 1424 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4408, 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2.737811095, 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000621100 2024-03-28T02:21:35.246 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4130, 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.184031296, 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000044559 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 189, 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 189, 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 189, 2024-03-28T02:21:35.247 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.181141434, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000958420 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 72, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 32, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 7, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 25, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:35.248 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 22, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 2, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 2, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 2, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 274, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 218, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-03-28T02:21:35.249 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 4, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 22, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 1692, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 4289, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 4252, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 4, 2024-03-28T02:21:35.251 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 27 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 7, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 18, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 2, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 5, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 3, 2024-03-28T02:21:35.252 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-28T02:21:35.253 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 4398, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 4339, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 4339, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 59, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 3, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 19, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 18, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 18, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 1, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 1, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 1, 2024-03-28T02:21:35.254 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 15782528, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 15868284, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4361, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 419.675291076, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.096233728 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 25, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 28, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 3, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 20, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 20, 2024-03-28T02:21:35.255 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 0, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 32, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 37, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 5, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 2, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 19, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 17, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 59084, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 231804 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 249, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 189, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 38, 2024-03-28T02:21:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 7, 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.011248980, 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001606997 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 109, 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.019174538, 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000175913 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 22, 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.005266425, 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000239382 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 10, 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001292339, 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000129233 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8, 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.012009392, 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001501174 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 10, 2024-03-28T02:21:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.060684474, 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.006068447 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001202173, 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000601086 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-28T02:21:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004477634, 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002238817 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.016286184, 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.008143092 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4, 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.019287074, 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.004821768 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001626304, 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000813152 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8, 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.020379254, 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002547406 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.008206663, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002735554 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 2, 2024-03-28T02:21:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 6, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 4, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 2, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 27, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 13, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 598, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:35.269 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 383535, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 160, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 968, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 11, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 183, 2024-03-28T02:21:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 123947, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 1008, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-28T02:21:35.272 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 9717, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 11796229, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 37, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 9717, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 9717, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 464.711338629, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.047824569 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 9717, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 9930 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 9717, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 3, 2024-03-28T02:21:35.273 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 9714, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 52, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 38, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 3, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 4364, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 62, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 5129, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:35.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 69, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:35.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-28T02:21:35.276 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 102, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 18, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 12, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 12, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 198, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 15 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-28T02:21:35.277 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 3585, 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 1, 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 4613, 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 3, 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.278 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 17961, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 5218026, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 17961, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 17961, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 5218026, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.279 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 9717, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 11796556, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 9717, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 11796556, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.280 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 9717, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 9717, 2024-03-28T02:21:35.281 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 9717, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 9717, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 3, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 327, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:35.282 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 3, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 3, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 327, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x55991cba8500": { 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 4398, 2024-03-28T02:21:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 11673980, 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 4398, 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 4361, 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 11673980, 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:35.284 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-28T02:21:35.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:35 smithi106 ceph-mon[31965]: pgmap v320: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:35.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:35 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:35 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:35 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:35 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:35 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:21:35.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:35 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:21:35.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:35 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:35.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:35 smithi178 ceph-mon[36521]: pgmap v320: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:35 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:35 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:35 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:35 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:35 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:21:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:35 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:21:35.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:35 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:35 smithi153 ceph-mon[36228]: pgmap v320: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:35.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:35 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:35 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:35 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:35 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T02:21:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:35 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T02:21:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:35 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T02:21:35.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:35 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' 2024-03-28T02:21:36.044 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-28T02:21:36.044 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-28T02:21:36.044 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=7 2024-03-28T02:21:36.044 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-28T02:21:36.044 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-28T02:21:36.044 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-28T02:21:36.044 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-28T02:21:36.044 DEBUG:tasks.check_counter:Getting stats from e 2024-03-28T02:21:36.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-28T02:21:36.309 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:21:36.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:36 smithi153 ceph-mon[36228]: Adjusting osd_memory_target on smithi106 to 176.2M 2024-03-28T02:21:36.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:36 smithi153 ceph-mon[36228]: Unable to set osd_memory_target on smithi106 to 184782438: error parsing value: Value '184782438' is below minimum 939524096 2024-03-28T02:21:36.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:36 smithi178 ceph-mon[36521]: Adjusting osd_memory_target on smithi106 to 176.2M 2024-03-28T02:21:36.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:36 smithi178 ceph-mon[36521]: Unable to set osd_memory_target on smithi106 to 184782438: error parsing value: Value '184782438' is below minimum 939524096 2024-03-28T02:21:37.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:36 smithi106 ceph-mon[31965]: Adjusting osd_memory_target on smithi106 to 176.2M 2024-03-28T02:21:37.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:36 smithi106 ceph-mon[31965]: Unable to set osd_memory_target on smithi106 to 184782438: error parsing value: Value '184782438' is below minimum 939524096 2024-03-28T02:21:37.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:37 smithi153 ceph-mon[36228]: pgmap v321: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:37.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:37 smithi178 ceph-mon[36521]: pgmap v321: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:38.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:37 smithi106 ceph-mon[31965]: pgmap v321: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:38.138 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-28T02:21:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 370, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 551, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 118849, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 132048, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 9, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.037769401, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.016746852, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.016581391, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.002749936, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 555, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.031936978, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000057544 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.139 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 834, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000151335, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000181 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 337, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 322, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1516376, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 161892, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 7, 2024-03-28T02:21:38.140 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.039166461, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.010402462, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.036438809, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.004070144, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 321, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.037526773, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000116905 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 67, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000011248, 2024-03-28T02:21:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000167 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1493472, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 136608 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 71, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 239, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 27281, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 516685, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 16, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 7, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.013705884, 2024-03-28T02:21:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.006019570, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.006349049, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000129745, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 231, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.022227849, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000096224 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 84, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000013758, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000163 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 5488, 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 444160 2024-03-28T02:21:38.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 43, 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001664654, 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000038712 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-28T02:21:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000005181, 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000002590 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 4, 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 3, 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3, 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000728975, 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000242991 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.145 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 1, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 11, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 13, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 13, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 12, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 1, 2024-03-28T02:21:38.146 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 2, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 10, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 5, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 1, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 2, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-28T02:21:38.147 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 4, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:38.148 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 3, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 2, 2024-03-28T02:21:38.149 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 4, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-28T02:21:38.150 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 4, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 4, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-03-28T02:21:38.151 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 2, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4195424, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.005215072, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001738357 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.152 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 16, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 16, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 0, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 15, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 16, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 1, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 13, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 14, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 1, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 2, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 2, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 50856, 2024-03-28T02:21:38.153 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 223612 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 5, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 4, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 4, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-03-28T02:21:38.154 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000212372, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000212372 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000516603, 2024-03-28T02:21:38.155 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000258301 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.158 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-28T02:21:38.158 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.158 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.158 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.158 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-28T02:21:38.158 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.158 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.159 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.160 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.161 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.162 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-28T02:21:38.164 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 0, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 1, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 1, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 0, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 2, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 2, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 85, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:38.165 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:38.166 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:38.167 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 246721, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 81, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 183, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:38.168 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 63880, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 247, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 41, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 11567, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 33, 2024-03-28T02:21:38.169 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 41, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 41, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.207996164, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.005073077 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 41, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 65 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 41, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 0, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 41, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-28T02:21:38.170 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 10, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 3, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 7, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 6, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 0, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 20, 2024-03-28T02:21:38.171 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-28T02:21:38.172 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 102, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 13, 2024-03-28T02:21:38.173 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 8, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 8, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 17, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 2 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-28T02:21:38.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 774, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1575442, 2024-03-28T02:21:38.175 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 774, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 774, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1575442, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-28T02:21:38.176 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 41, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 11567, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 41, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 11567, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:38.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 41, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 41, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 41, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 41, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.178 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-28T02:21:38.179 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x565029694500": { 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 4, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1120, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 4, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:38.180 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 3, 2024-03-28T02:21:38.181 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1120, 2024-03-28T02:21:38.181 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:38.181 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:38.181 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:38.181 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:38.181 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:38.181 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:38.181 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-28T02:21:38.971 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-28T02:21:38.971 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-28T02:21:38.971 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-28T02:21:38.971 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=3 2024-03-28T02:21:38.971 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-28T02:21:38.971 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-28T02:21:38.971 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T02:21:38.971 DEBUG:tasks.check_counter:Getting stats from h 2024-03-28T02:21:38.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-28T02:21:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:21:39.674 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:39 smithi153 ceph-mon[36228]: pgmap v322: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:39.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:39 smithi178 ceph-mon[36521]: pgmap v322: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:40.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:39 smithi106 ceph-mon[31965]: pgmap v322: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:40.912 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-28T02:21:40.912 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:40.912 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1304, 2024-03-28T02:21:40.912 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1286, 2024-03-28T02:21:40.912 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 6024314, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 465896, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 5, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 1, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.110305506, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.034532432, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.053443987, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.026358236, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1285, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.031630844, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000024615 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.913 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1498464, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 130720 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1011, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1171, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 3542338, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 761264, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 7, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.059834171, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.018382692, 2024-03-28T02:21:40.914 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.016385435, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.019578206, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1168, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.031615145, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000027067 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 440336 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.915 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 288, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 288, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 4267618, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 90767, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 4, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.023083454, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.005314137, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.008171885, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.007756826, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 287, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.007720011, 2024-03-28T02:21:40.916 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000026898 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-28T02:21:40.917 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3912, 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.075537555, 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000019309 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.918 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-03-28T02:21:40.919 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-28T02:21:40.920 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-28T02:21:40.920 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 26, 2024-03-28T02:21:40.920 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 8, 2024-03-28T02:21:40.920 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 18, 2024-03-28T02:21:40.920 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:40.920 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 15, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 2, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-28T02:21:40.921 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 4, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-28T02:21:40.923 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:40.924 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 4398, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-28T02:21:40.925 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 14, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 15782528, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 15868284, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 15868148, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 4398 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 17, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 20, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 3, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 12, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 12, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 0, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 26, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 26, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-03-28T02:21:40.927 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 50416, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-28T02:21:40.928 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.929 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.930 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-28T02:21:40.931 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.932 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.932 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.932 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-28T02:21:40.932 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.932 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.932 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.933 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.934 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 2, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 6, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 4, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 4, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 3634, 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 190663, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 59, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 183, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 86618, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 594, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 2347, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 26, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 4, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 2347, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2347, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2.682950415, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001143140 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2347, 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2348 2024-03-28T02:21:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 2347, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 2347, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 1500, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 844, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-03-28T02:21:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 2, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-28T02:21:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-28T02:21:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 102, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 13, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 5, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 5, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-03-28T02:21:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2599, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 13584602, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2599, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2599, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 13584602, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2347, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 11673990, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2347, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 11673990, 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2347, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2347, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2347, 2024-03-28T02:21:40.952 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2347, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:40.953 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x557a4fb7c6e0": { 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:40.954 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:40.955 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-28T02:21:41.608 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-28T02:21:41.608 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-28T02:21:41.608 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-28T02:21:41.608 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-03-28T02:21:41.608 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-28T02:21:41.608 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-28T02:21:41.608 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T02:21:41.608 DEBUG:tasks.check_counter:Getting stats from k 2024-03-28T02:21:41.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-28T02:21:41.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:41 smithi178 ceph-mon[36521]: pgmap v323: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:41.913 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T02:21:41.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:41 smithi153 ceph-mon[36228]: pgmap v323: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:42.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:41 smithi106 ceph-mon[31965]: pgmap v323: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:43.661 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:43 smithi153 ceph-mon[36228]: pgmap v324: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:43.734 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-28T02:21:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 248, 2024-03-28T02:21:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 237, 2024-03-28T02:21:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1472482, 2024-03-28T02:21:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 134004, 2024-03-28T02:21:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 5, 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 1, 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.037005346, 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.009079464, 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.033556098, 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.004071080, 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 236, 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.006981779, 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000029583 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1471648, 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 133680 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 695, 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 860, 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 170820, 2024-03-28T02:21:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 663195, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 7, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.054775609, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.017580169, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.012450902, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.018845384, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 857, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.031835982, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000037148 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 442816 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1515, 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1515, 2024-03-28T02:21:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 457116, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 481004, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 4, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.097357046, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.032548536, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.024846754, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.032053318, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1514, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.034608629, 2024-03-28T02:21:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000022859 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-28T02:21:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3681, 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.076347543, 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000020740 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-03-28T02:21:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-28T02:21:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 1, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-03-28T02:21:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-28T02:21:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 9, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-03-28T02:21:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4196879, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 4196738, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 9 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 3, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 3, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 0, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 1, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 1, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 0, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 0, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 0, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-03-28T02:21:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 47512, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-28T02:21:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-28T02:21:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-28T02:21:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-28T02:21:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-28T02:21:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-28T02:21:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 4, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 3, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 3, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 5144, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 190201, 2024-03-28T02:21:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 53, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 183, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 8320, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 13, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-28T02:21:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 2205, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 21, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 3, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 2205, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2205, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 1.190978277, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000540126 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2205, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2206 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 2205, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 2205, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 1443, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 760, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-28T02:21:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 1, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-28T02:21:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 102, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 13, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 5, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 5, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-03-28T02:21:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2454, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1863889, 2024-03-28T02:21:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2454, 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2454, 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1863889, 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2205, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2580, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2205, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2580, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2205, 2024-03-28T02:21:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2205, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2205, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2205, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x55eaebab66e0": { 2024-03-28T02:21:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-28T02:21:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:43.786 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-28T02:21:43.786 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-28T02:21:43.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:43 smithi178 ceph-mon[36521]: pgmap v324: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:44.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:43 smithi106 ceph-mon[31965]: pgmap v324: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:44.576 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-28T02:21:44.577 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-28T02:21:44.577 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-03-28T02:21:44.577 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-03-28T02:21:44.577 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-03-28T02:21:44.577 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-28T02:21:44.577 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T02:21:44.577 DEBUG:tasks.check_counter:Getting stats from c 2024-03-28T02:21:44.577 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-28T02:21:44.903 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:21:45.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:45 smithi178 ceph-mon[36521]: pgmap v325: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:45.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:45 smithi153 ceph-mon[36228]: pgmap v325: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:46.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:45 smithi106 ceph-mon[31965]: pgmap v325: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:46.992 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 1008, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 990, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 1753200, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 372534, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 5, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 1, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.091494707, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.025759522, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.047061204, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.023351802, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 989, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.027434480, 2024-03-28T02:21:46.993 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000027739 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 1498848, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 131808 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 4, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 166, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 876, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 444586, 2024-03-28T02:21:46.994 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 6, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 4, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.008906849, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.004869006, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.002533169, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.000073551, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 163, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.010571399, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000064855 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.995 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 443888 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 1397, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 1397, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 360278, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 443429, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 4, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 3, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.077227279, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.022727673, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.020042374, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.030155201, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 1396, 2024-03-28T02:21:46.996 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.031004804, 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000022209 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "cct": { 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "total_workers": 1, 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "unhealthy_workers": 0 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.997 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-MDSRank": { 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 3663, 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.076600060, 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000020911 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:46.998 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "mds": { 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "request": 0, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "reply": 0, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "reply_latency": { 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_reply": 0, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "forward": 0, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_commit": 0, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_split": 0, 2024-03-28T02:21:46.999 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_merge": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_top": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_bottom": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pinned": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_expired": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_with_caps": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "caps": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "subtrees": 1, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_hit": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_forward": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_discover": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:47.000 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_lock": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "load_cent": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "q": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "exported": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "exported_inodes": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "imported": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "imported_inodes": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rfiles": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rbytes": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rsnaps": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:47.001 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_grant": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps": 0, 2024-03-28T02:21:47.002 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps_dirty": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_cap_release": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "process_request_cap_release": 0 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache": { 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update_receipt": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_try_discover": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_send_discover": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_handle_discover": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_created": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_enqueued": 0, 2024-03-28T02:21:47.003 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_migrated": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_started": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_completed": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_flush": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_inodestats": 0 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log": { 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "evlrg": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "evadd": 0, 2024-03-28T02:21:47.004 INFO:teuthology.orchestra.run.smithi178.stdout: "evex": 0, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "evtrm": 0, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "ev": 6, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "evexg": 0, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "evexd": 0, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "segadd": 0, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "segex": 0, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "segtrm": 0, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "seg": 2, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "segmjr": 2, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "segexg": 0, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "segexd": 0, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "expos": 4194304, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "wrpos": 4196123, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "rdpos": 4195982, 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "jlat": { 2024-03-28T02:21:47.005 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "replayed": 6 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_mem": { 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "ino": 3, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "ino+": 3, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "ino-": 0, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "dir": 1, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "dir+": 1, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "dir-": 0, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "dn": 0, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "dn+": 0, 2024-03-28T02:21:47.006 INFO:teuthology.orchestra.run.smithi178.stdout: "dn-": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "cap": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "cap+": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "cap-": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "rss": 47704, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "heap": 215420 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_server": { 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_client_request": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_request": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_session": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_peer_request": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "req_create_latency": { 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.007 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getattr_latency": { 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "req_link_latency": { 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.008 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookup_latency": { 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupino_latency": { 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.009 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupname_latency": { 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lssnap_latency": { 2024-03-28T02:21:47.010 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.011 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.011 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.011 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.011 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mkdir_latency": { 2024-03-28T02:21:47.011 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.011 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.011 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.011 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.011 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mknod_latency": { 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mksnap_latency": { 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "req_open_latency": { 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "req_readdir_latency": { 2024-03-28T02:21:47.012 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rename_latency": { 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.013 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmdir_latency": { 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.014 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setattr_latency": { 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.015 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setlayout_latency": { 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setxattr_latency": { 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "req_symlink_latency": { 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.016 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "req_unlink_latency": { 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sessions": { 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "session_count": 2, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "session_add": 3, 2024-03-28T02:21:47.017 INFO:teuthology.orchestra.run.smithi178.stdout: "session_remove": 1, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_open": 3, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_stale": 0, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "total_load": 0, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "average_load": 0, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "avg_session_uptime": 2862, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "mempool": { 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:47.018 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:47.019 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_items": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_bytes": 181353, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_items": 48, 2024-03-28T02:21:47.020 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bytes": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_items": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_items": 183, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_items": 0, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_bytes": 8320, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_items": 13, 2024-03-28T02:21:47.021 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_items": 0, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_items": 0 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter": { 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "op_active": 0, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "op_laggy": 0, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send": 2152, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send_bytes": 21, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "op_resend": 3, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "op_reply": 2152, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "op_latency": { 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2152, 2024-03-28T02:21:47.022 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 2.833950438, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.001316891 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "op_inflight": 0, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "oplen_avg": { 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2152, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 2153 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "op": 2152, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "op_r": 2152, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "op_w": 0, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "op_rmw": 0, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "op_pg": 0, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_stat": 1385, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_create": 0, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_read": 765, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_write": 0, 2024-03-28T02:21:47.023 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writefull": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writesame": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_append": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_zero": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_truncate": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_delete": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_mapext": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_getxattr": 1, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_setxattr": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_call": 0, 2024-03-28T02:21:47.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_watch": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_notify": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_other": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_active": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_send": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_resend": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_ping": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_active": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_send": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_resend": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_active": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_send": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_resend": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_active": 0, 2024-03-28T02:21:47.025 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_send": 0, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_resend": 0, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "command_active": 0, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "command_send": 0, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "command_resend": 0, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "map_epoch": 102, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "map_full": 0, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "map_inc": 13, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_sessions": 4, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_open": 4, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_close": 0, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_laggy": 0, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_wr": 0, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_rd": 2, 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_del": 0 2024-03-28T02:21:47.026 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "oft": { 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_objs": 0, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_updates": 0, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_removes": 0 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "osd-slow-ops": { 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_ops_count": 0 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "purge_queue": { 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing": 0, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_high_water": 0, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed_ops": 0, 2024-03-28T02:21:47.027 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed": 0, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 2405, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 1882088, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 2405, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2405, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 1882088, 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:47.028 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 2152, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 1824, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2152, 2024-03-28T02:21:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 1824, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 1024, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 2152, 2024-03-28T02:21:47.030 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 2152, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2152, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 2152, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:47.031 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle-0x564dc5e9e6e0": { 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:47.032 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:47.033 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-28T02:21:47.699 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:47 smithi178 ceph-mon[36521]: pgmap v326: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:47.729 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-28T02:21:47.729 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-28T02:21:47.729 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-28T02:21:47.729 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-28T02:21:47.729 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-28T02:21:47.729 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-28T02:21:47.729 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T02:21:47.729 DEBUG:tasks.check_counter:Getting stats from f 2024-03-28T02:21:47.729 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-28T02:21:47.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:47 smithi153 ceph-mon[36228]: pgmap v326: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:48.022 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:21:48.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:47 smithi106 ceph-mon[31965]: pgmap v326: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:49.807 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:49 smithi178 ceph-mon[36521]: pgmap v327: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:49.807 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-28T02:21:49.807 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:49.807 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 87, 2024-03-28T02:21:49.807 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 90, 2024-03-28T02:21:49.807 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 29395, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 79912, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 16, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 2, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.011538230, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.002413314, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.006193946, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.001675303, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 88, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.021371522, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000242858 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 105, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000018571, 2024-03-28T02:21:49.808 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000176 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 1856, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 1712 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 72, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 234, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 27963, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 477493, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 11, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 7, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.018685853, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.007573137, 2024-03-28T02:21:49.809 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.006447195, 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.001266192, 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 232, 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.043276009, 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000186534 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 103, 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000027284, 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000264 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 4272, 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 445488 2024-03-28T02:21:49.810 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 667, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 844, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 1627123, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 396067, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 8, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 5, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.077679584, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.025662046, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.060514248, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.006861632, 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:49.811 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 843, 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.044272556, 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000052517 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 926, 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000174305, 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000188 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 1502752, 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 143008 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "cct": { 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "total_workers": 1, 2024-03-28T02:21:49.812 INFO:teuthology.orchestra.run.smithi178.stdout: "unhealthy_workers": 0 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-MDSRank": { 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 54, 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.002389102, 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000044242 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2, 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000010098, 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000005049 2024-03-28T02:21:49.813 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "mds": { 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "request": 0, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "reply": 0, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "reply_latency": { 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_reply": 0, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "forward": 0, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_commit": 11, 2024-03-28T02:21:49.814 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_split": 0, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_merge": 0, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes": 14, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_top": 1, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_bottom": 13, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pinned": 12, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_expired": 0, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_with_caps": 0, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "caps": 0, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "subtrees": 2, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse": 4, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_hit": 2, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_forward": 0, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_discover": 2, 2024-03-28T02:21:49.815 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_lock": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "load_cent": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "q": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "exported": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "exported_inodes": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "imported": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "imported_inodes": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rfiles": 4, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rbytes": 234, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rsnaps": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:49.816 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_grant": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:49.817 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps": 0, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps_dirty": 0, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_cap_release": 0, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "process_request_cap_release": 0 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache": { 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update": 0, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update_receipt": 6, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_try_discover": 4, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_send_discover": 5, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_handle_discover": 0, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays": 0, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_created": 0, 2024-03-28T02:21:49.818 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_enqueued": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_migrated": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_started": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_completed": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_flush": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_inodestats": 0 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log": { 2024-03-28T02:21:49.819 INFO:teuthology.orchestra.run.smithi178.stdout: "evlrg": 0, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "evadd": 9, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "evex": 0, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "evtrm": 0, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "ev": 9, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "evexg": 0, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "evexd": 0, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "segadd": 2, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "segex": 0, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "segtrm": 0, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "seg": 2, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "segmjr": 2, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "segexg": 0, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "segexd": 0, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "expos": 4194304, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "wrpos": 4196879, 2024-03-28T02:21:49.820 INFO:teuthology.orchestra.run.smithi178.stdout: "rdpos": 0, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "jlat": { 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 8, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.013810477, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.001726309 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "replayed": 0 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_mem": { 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "ino": 17, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "ino+": 17, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "ino-": 0, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "dir": 16, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "dir+": 16, 2024-03-28T02:21:49.821 INFO:teuthology.orchestra.run.smithi178.stdout: "dir-": 0, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "dn": 14, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "dn+": 14, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "dn-": 0, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "cap": 0, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "cap+": 0, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "cap-": 0, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "rss": 50100, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "heap": 223612 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_server": { 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_client_request": 0, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_request": 0, 2024-03-28T02:21:49.822 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_session": 20, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_peer_request": 0, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "req_create_latency": { 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getattr_latency": { 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.823 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "req_link_latency": { 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookup_latency": { 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.824 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupino_latency": { 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupname_latency": { 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:49.825 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lssnap_latency": { 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mkdir_latency": { 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.826 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mknod_latency": { 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mksnap_latency": { 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.827 INFO:teuthology.orchestra.run.smithi178.stdout: "req_open_latency": { 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "req_readdir_latency": { 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rename_latency": { 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.828 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmdir_latency": { 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.829 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setattr_latency": { 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.830 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setlayout_latency": { 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setxattr_latency": { 2024-03-28T02:21:49.831 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "req_symlink_latency": { 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "req_unlink_latency": { 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.832 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sessions": { 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "session_count": 1, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "session_add": 4, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "session_remove": 3, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_open": 1, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_stale": 0, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "total_load": 0, 2024-03-28T02:21:49.833 INFO:teuthology.orchestra.run.smithi178.stdout: "average_load": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "avg_session_uptime": 584, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "mempool": { 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:49.834 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:49.835 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_items": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_bytes": 262711, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_items": 90, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bytes": 0, 2024-03-28T02:21:49.836 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_items": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_items": 183, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_items": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_bytes": 67936, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_items": 251, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_items": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:49.837 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_items": 0 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter": { 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "op_active": 0, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "op_laggy": 0, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send": 46, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send_bytes": 13057, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "op_resend": 33, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "op_reply": 46, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "op_latency": { 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 46, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.277995064, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.006043370 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "op_inflight": 0, 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "oplen_avg": { 2024-03-28T02:21:49.838 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 46, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 68 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "op": 46, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "op_r": 0, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "op_w": 46, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "op_rmw": 0, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "op_pg": 0, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_stat": 0, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_create": 10, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_read": 0, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_write": 8, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writefull": 9, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writesame": 0, 2024-03-28T02:21:49.839 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_append": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_zero": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_truncate": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_delete": 6, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_mapext": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_getxattr": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_setxattr": 20, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_call": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_watch": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_notify": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls": 0, 2024-03-28T02:21:49.840 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_other": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_active": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_send": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_resend": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_ping": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_active": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_send": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_resend": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_active": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_send": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_resend": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_active": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_send": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_resend": 0, 2024-03-28T02:21:49.841 INFO:teuthology.orchestra.run.smithi178.stdout: "command_active": 0, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "command_send": 0, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "command_resend": 0, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "map_epoch": 102, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "map_full": 0, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "map_inc": 13, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_sessions": 8, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_open": 8, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_close": 0, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_laggy": 0, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_wr": 14, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_rd": 0, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_del": 1 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "oft": { 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_objs": 1, 2024-03-28T02:21:49.842 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_updates": 0, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_removes": 0 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "osd-slow-ops": { 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_ops_count": 0 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "purge_queue": { 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing": 0, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_high_water": 0, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed_ops": 0, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed": 0, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.843 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 822, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 1592390, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 822, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 822, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 1592390, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.844 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 46, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 13057, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 46, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 13057, 2024-03-28T02:21:49.845 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 1024, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:49.846 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 46, 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 46, 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 46, 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 46, 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-28T02:21:49.847 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle-0x559cb8a1c500": { 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 9, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 2575, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 9, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 8, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 2575, 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:49.849 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:49.850 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:49.850 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:49.850 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:49.850 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:49.850 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-28T02:21:49.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:49 smithi153 ceph-mon[36228]: pgmap v327: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:50.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:49 smithi106 ceph-mon[31965]: pgmap v327: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:50.681 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-28T02:21:50.681 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-28T02:21:50.681 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-28T02:21:50.682 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=6 2024-03-28T02:21:50.682 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-28T02:21:50.682 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-28T02:21:50.682 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T02:21:50.682 DEBUG:tasks.check_counter:Getting stats from i 2024-03-28T02:21:50.682 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-28T02:21:50.948 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:21:51.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:51 smithi178 ceph-mon[36521]: pgmap v328: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:51.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:51 smithi153 ceph-mon[36228]: pgmap v328: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:52.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:51 smithi106 ceph-mon[31965]: pgmap v328: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:52.842 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-28T02:21:52.842 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:52.842 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 412, 2024-03-28T02:21:52.842 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 611, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 134044, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 352447, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 11, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 3, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.041004858, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.015876628, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.019973825, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.003396606, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 614, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.038311691, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000062396 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 936, 2024-03-28T02:21:52.843 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000172369, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000184 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 4592, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 5600 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 34, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 232, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 7259, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 505638, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 17, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 8, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.017555631, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.006791677, 2024-03-28T02:21:52.844 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.008665294, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.000251998, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 223, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.026914246, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000120691 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 64, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000024830, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000387 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 2176, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 448352 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:52.845 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 390, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 369, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 1548729, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 187524, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 7, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 5, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.044723644, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.012767619, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.040084911, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.004587305, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 370, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.024940133, 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000067405 2024-03-28T02:21:52.846 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 169, 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000030856, 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000182 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 1505824, 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 137232 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "cct": { 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "total_workers": 1, 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "unhealthy_workers": 0 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-MDSRank": { 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-28T02:21:52.847 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 52, 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.001760008, 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000033846 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2, 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000008634, 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000004317 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "mds": { 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "request": 5, 2024-03-28T02:21:52.848 INFO:teuthology.orchestra.run.smithi178.stdout: "reply": 4, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "reply_latency": { 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 4, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000849290, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000212322 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_reply": 0, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "forward": 1, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_commit": 11, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_split": 0, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_merge": 0, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes": 14, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_top": 1, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_bottom": 13, 2024-03-28T02:21:52.849 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pinned": 13, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_expired": 0, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_with_caps": 2, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "caps": 2, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "subtrees": 2, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse": 10, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_hit": 7, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_forward": 1, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_discover": 2, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_lock": 0, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "load_cent": 0, 2024-03-28T02:21:52.850 INFO:teuthology.orchestra.run.smithi178.stdout: "q": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "exported": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "exported_inodes": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "imported": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "imported_inodes": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rfiles": 4, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rbytes": 234, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rsnaps": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:52.851 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_grant": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps_dirty": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_cap_release": 0, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "process_request_cap_release": 0 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache": { 2024-03-28T02:21:52.852 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update_receipt": 8, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_try_discover": 5, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_send_discover": 6, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_handle_discover": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_created": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_enqueued": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_migrated": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_started": 0, 2024-03-28T02:21:52.853 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_completed": 0, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_flush": 0, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_inodestats": 0 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log": { 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "evlrg": 0, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "evadd": 6, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "evex": 0, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "evtrm": 0, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "ev": 6, 2024-03-28T02:21:52.854 INFO:teuthology.orchestra.run.smithi178.stdout: "evexg": 0, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "evexd": 0, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "segadd": 2, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "segex": 0, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "segtrm": 0, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "seg": 2, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "segmjr": 2, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "segexg": 0, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "segexd": 0, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "expos": 4194304, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "wrpos": 4196123, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "rdpos": 0, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "jlat": { 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 5, 2024-03-28T02:21:52.855 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.008057004, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.001611400 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "replayed": 0 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_mem": { 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "ino": 17, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "ino+": 17, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "ino-": 0, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "dir": 16, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "dir+": 17, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "dir-": 1, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "dn": 14, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "dn+": 14, 2024-03-28T02:21:52.856 INFO:teuthology.orchestra.run.smithi178.stdout: "dn-": 0, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "cap": 2, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "cap+": 4, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "cap-": 2, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "rss": 52128, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "heap": 223612 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_server": { 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_client_request": 5, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_request": 5, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_session": 21, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_peer_request": 0, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "req_create_latency": { 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.857 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getattr_latency": { 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 1, 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000300183, 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000300183 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "req_link_latency": { 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.858 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookup_latency": { 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 1, 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000318403, 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000318403 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupino_latency": { 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2, 2024-03-28T02:21:52.859 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000230704, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000115352 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupname_latency": { 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.860 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lssnap_latency": { 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mkdir_latency": { 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.861 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mknod_latency": { 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mksnap_latency": { 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.862 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "req_open_latency": { 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "req_readdir_latency": { 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rename_latency": { 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.863 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmdir_latency": { 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.864 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setattr_latency": { 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.865 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setlayout_latency": { 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.866 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setxattr_latency": { 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "req_symlink_latency": { 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "req_unlink_latency": { 2024-03-28T02:21:52.867 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sessions": { 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "session_count": 2, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "session_add": 3, 2024-03-28T02:21:52.868 INFO:teuthology.orchestra.run.smithi178.stdout: "session_remove": 1, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_open": 2, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_stale": 0, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "total_load": 3, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "average_load": 1, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "avg_session_uptime": 609, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "mempool": { 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:52.869 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:52.870 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_items": 0, 2024-03-28T02:21:52.871 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_bytes": 279383, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_items": 98, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_bytes": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_items": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bytes": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_items": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:52.872 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_items": 183, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_items": 0, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_bytes": 68720, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_items": 261, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_items": 0, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_items": 0 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter": { 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "op_active": 0, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "op_laggy": 0, 2024-03-28T02:21:52.873 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send": 45, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send_bytes": 12562, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "op_resend": 33, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "op_reply": 45, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "op_latency": { 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 45, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.227995964, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.005066576 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "op_inflight": 0, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "oplen_avg": { 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 45, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 70 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "op": 45, 2024-03-28T02:21:52.874 INFO:teuthology.orchestra.run.smithi178.stdout: "op_r": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "op_w": 45, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "op_rmw": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "op_pg": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_stat": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_create": 10, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_read": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_write": 5, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writefull": 8, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writesame": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_append": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_zero": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_truncate": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_delete": 6, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_mapext": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:52.875 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_getxattr": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_setxattr": 20, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_call": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_watch": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_notify": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_other": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_active": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_send": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_resend": 0, 2024-03-28T02:21:52.876 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_ping": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_active": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_send": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_resend": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_active": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_send": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_resend": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_active": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_send": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_resend": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "command_active": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "command_send": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "command_resend": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "map_epoch": 102, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "map_full": 0, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "map_inc": 13, 2024-03-28T02:21:52.877 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_sessions": 9, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_open": 9, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_close": 0, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_laggy": 0, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_wr": 20, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_rd": 0, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_del": 1 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "oft": { 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_objs": 1, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_updates": 0, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_removes": 0 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "osd-slow-ops": { 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_ops_count": 0 2024-03-28T02:21:52.878 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "purge_queue": { 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing": 0, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_high_water": 0, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed_ops": 0, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed": 0, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:52.879 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 832, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 1597203, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 832, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 832, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 1597203, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-28T02:21:52.880 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 45, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 12562, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 45, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 12562, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:52.881 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 1024, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 45, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 45, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 45, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 45, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.882 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:52.883 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle-0x556b4cd0a500": { 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 6, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 1819, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 6, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 5, 2024-03-28T02:21:52.884 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 1819, 2024-03-28T02:21:52.885 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:52.885 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:52.885 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:52.885 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:52.885 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:52.885 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:52.885 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-28T02:21:53.670 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-28T02:21:53.670 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-28T02:21:53.670 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-28T02:21:53.670 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=8 2024-03-28T02:21:53.670 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=0 2024-03-28T02:21:53.670 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-28T02:21:53.670 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T02:21:53.670 DEBUG:tasks.check_counter:Getting stats from l 2024-03-28T02:21:53.671 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d222a03c-eca6-11ee-95d0-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-28T02:21:53.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:53 smithi178 ceph-mon[36521]: pgmap v329: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:53.926 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T02:21:53.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:53 smithi153 ceph-mon[36228]: pgmap v329: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:54.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:53 smithi106 ceph-mon[31965]: pgmap v329: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:55.752 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:55 smithi178 ceph-mon[36521]: pgmap v330: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:55.776 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 351, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 344, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 1511645, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 212908, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 15, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 6, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.047906118, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.012236847, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.043180749, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.004872877, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 338, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.019819384, 2024-03-28T02:21:55.777 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000058637 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 95, 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000019306, 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000203 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 1487968, 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 141952 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 346, 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 703, 2024-03-28T02:21:55.778 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 110196, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 671073, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 10, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 5, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.042924406, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.017801363, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.020384602, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.002789955, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 706, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.043224430, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000061224 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 849, 2024-03-28T02:21:55.779 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000153898, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000181 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 446816 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 82, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 112, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 28700, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 40272, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 9, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 6, 2024-03-28T02:21:55.780 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.009291695, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.003123634, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.005641432, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.000678422, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 110, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.021629402, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000196630 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 129, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000024662, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000191 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.781 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 3488, 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 11056 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "cct": { 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "total_workers": 1, 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "unhealthy_workers": 0 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-MDSRank": { 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 37, 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.001610408, 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000043524 2024-03-28T02:21:55.782 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-PurgeQueue": { 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2, 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000004959, 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000002479 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "mds": { 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "request": 6, 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "reply": 3, 2024-03-28T02:21:55.783 INFO:teuthology.orchestra.run.smithi178.stdout: "reply_latency": { 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 3, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000646319, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000215439 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_reply": 0, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "forward": 3, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_complete": 0, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_keys": 0, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_commit": 11, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_split": 0, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_merge": 0, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes": 13, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_top": 0, 2024-03-28T02:21:55.784 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_bottom": 13, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pin_tail": 0, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pinned": 12, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_expired": 0, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_with_caps": 1, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "caps": 1, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "subtrees": 2, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse": 9, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_hit": 7, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_forward": 0, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_discover": 1, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_dir_fetch": 0, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_remote_ino": 0, 2024-03-28T02:21:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_lock": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "load_cent": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "q": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "exported": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "exported_inodes": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "imported": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "imported_inodes": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_dir_fetch": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_backtrace_fetch": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_peer_discover": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rfiles": 4, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rbytes": 234, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rsnaps": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T02:21:55.786 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_set_tag": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_inotable_repaired": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_inodes": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_file_inodes": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_inode_file_caps": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_grant": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps_dirty": 0, 2024-03-28T02:21:55.787 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_cap_release": 0, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "process_request_cap_release": 0 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache": { 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update": 0, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update_receipt": 3, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_try_discover": 2, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_send_discover": 3, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_handle_discover": 0, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays": 0, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_delayed": 0, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_enqueuing": 0, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_created": 0, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_enqueued": 0, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_reintegrated": 0, 2024-03-28T02:21:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_migrated": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_processing": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_enqueued": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_prioritized": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_started": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_completed": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_exportdir": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_flush": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragmentdir": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragstats": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_inodestats": 0 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log": { 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "evlrg": 0, 2024-03-28T02:21:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "evadd": 5, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "evex": 0, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "evtrm": 0, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "ev": 5, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "evexg": 0, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "evexd": 0, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "segadd": 2, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "segex": 0, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "segtrm": 0, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "seg": 2, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "segmjr": 2, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "segexg": 0, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "segexd": 0, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "expos": 4194304, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "wrpos": 4195815, 2024-03-28T02:21:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "rdpos": 0, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "jlat": { 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 4, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.006571788, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.001642947 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "replayed": 0 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_mem": { 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "ino": 16, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "ino+": 16, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "ino-": 0, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "dir": 15, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "dir+": 16, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "dir-": 1, 2024-03-28T02:21:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "dn": 13, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "dn+": 13, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "dn-": 0, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "cap": 1, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "cap+": 1, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "cap-": 0, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "rss": 50608, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "heap": 223612 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_server": { 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_client_request": 6, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_server_request": 0, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_request": 6, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_session": 12, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_peer_request": 0, 2024-03-28T02:21:55.792 INFO:teuthology.orchestra.run.smithi178.stdout: "req_create_latency": { 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getattr_latency": { 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 3, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000646319, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000215439 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getfilelock_latency": { 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "req_link_latency": { 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.793 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookup_latency": { 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookuphash_latency": { 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupino_latency": { 2024-03-28T02:21:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupname_latency": { 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupparent_latency": { 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupsnap_latency": { 2024-03-28T02:21:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lssnap_latency": { 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mkdir_latency": { 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mknod_latency": { 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mksnap_latency": { 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "req_open_latency": { 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "req_readdir_latency": { 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rename_latency": { 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.798 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "req_renamesnap_latency": { 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "req_snapdiff_latency": { 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmdir_latency": { 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.799 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmsnap_latency": { 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmxattr_latency": { 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setattr_latency": { 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.800 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setdirlayout_latency": { 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setfilelock_latency": { 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setlayout_latency": { 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.801 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setxattr_latency": { 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "req_symlink_latency": { 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "req_unlink_latency": { 2024-03-28T02:21:55.802 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_revoke_eviction": 0, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_acquisition_throttle": 0, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getvxattr_latency": { 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sessions": { 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "session_count": 1, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "session_add": 2, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "session_remove": 1, 2024-03-28T02:21:55.803 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_open": 1, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_stale": 0, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "total_load": 2, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "average_load": 1, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "avg_session_uptime": 590, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "mdthresh_evicted": 0 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "mempool": { 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_bytes": 0, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_items": 0, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_items": 0, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_items": 0, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T02:21:55.804 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_items": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_bytes": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_items": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_bytes": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_items": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T02:21:55.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_items": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_bytes": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_items": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_bytes": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_items": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_bytes": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_items": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_items": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_items": 0, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_bytes": 248183, 2024-03-28T02:21:55.806 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_items": 89, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_bytes": 176, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_items": 2, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bytes": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_items": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_bytes": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_items": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_bytes": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_items": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_bytes": 9520, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_items": 183, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_items": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_bytes": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_items": 0, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_bytes": 64160, 2024-03-28T02:21:55.807 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_items": 238, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_bytes": 0, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_items": 0, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_bytes": 0, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_items": 0 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter": { 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "op_active": 0, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "op_laggy": 0, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send": 41, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send_bytes": 11903, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "op_resend": 33, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "op_reply": 41, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "op_latency": { 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 41, 2024-03-28T02:21:55.808 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.199996466, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.004877962 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "op_inflight": 0, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "oplen_avg": { 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 41, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 63 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "op": 41, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "op_r": 0, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "op_w": 41, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "op_rmw": 0, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "op_pg": 0, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_stat": 0, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_create": 10, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_read": 0, 2024-03-28T02:21:55.809 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_write": 4, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writefull": 8, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writesame": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_append": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_zero": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_truncate": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_delete": 6, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_mapext": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_sparse_read": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_clonerange": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_getxattr": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_setxattr": 20, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_cmpxattr": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_rmxattr": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_resetxattrs": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_call": 0, 2024-03-28T02:21:55.810 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_watch": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_notify": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls_filter": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_other": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_active": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_send": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_resend": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_ping": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_active": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_send": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_resend": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_active": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_send": 0, 2024-03-28T02:21:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_resend": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_active": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_send": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_resend": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "command_active": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "command_send": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "command_resend": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "map_epoch": 102, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "map_full": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "map_inc": 13, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_sessions": 9, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_open": 9, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_close": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_laggy": 0, 2024-03-28T02:21:55.812 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_wr": 14, 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_rd": 0, 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_del": 1 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "oft": { 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_objs": 1, 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_kv_pairs": 0, 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_updates": 0, 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_removes": 0 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "osd-slow-ops": { 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_ops_count": 0 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "purge_queue": { 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops": 0, 2024-03-28T02:21:55.813 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing": 0, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_high_water": 0, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed_ops": 0, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed": 0, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_item_in_journal": 0 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 775, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 1562650, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 775, 2024-03-28T02:21:55.814 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 775, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 1562650, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_bytes": { 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:55.815 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 41, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 11903, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 41, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 11903, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_ops": { 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 1024, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:55.816 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 41, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 41, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 41, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 41, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle": { 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:55.817 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:55.818 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle-0x5580e9b70500": { 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 5, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 1511, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 5, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 4, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 1511, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-28T02:21:55.819 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-28T02:21:55.820 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-28T02:21:55.820 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:55.820 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-28T02:21:55.820 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-28T02:21:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:55 smithi153 ceph-mon[36228]: pgmap v330: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:56.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:55 smithi106 ceph-mon[31965]: pgmap v330: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:56.785 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-28T02:21:56.785 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-28T02:21:56.786 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-28T02:21:56.786 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=3 2024-03-28T02:21:56.786 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-28T02:21:56.786 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-28T02:21:56.786 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T02:21:56.786 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-28T02:21:56.796 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-28T02:21:56.797 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:21:56.797 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T02:21:56.817 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-28T02:21:56.817 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T02:21:56.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:21:57.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.036+0000 7fe553f77640 1 -- 172.21.15.106:0/4132032306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101630 msgr2=0x7fe54c101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.036+0000 7fe553f77640 1 --2- 172.21.15.106:0/4132032306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101630 0x7fe54c101ab0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe548009930 tx=0x7fe54802bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.036+0000 7fe553f77640 1 -- 172.21.15.106:0/4132032306 shutdown_connections 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.036+0000 7fe553f77640 1 --2- 172.21.15.106:0/4132032306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe54c101ff0 0x7fe54c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.036+0000 7fe553f77640 1 --2- 172.21.15.106:0/4132032306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101630 0x7fe54c101ab0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.036+0000 7fe553f77640 1 --2- 172.21.15.106:0/4132032306 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe54c107640 0x7fe54c107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.036+0000 7fe553f77640 1 -- 172.21.15.106:0/4132032306 >> 172.21.15.106:0/4132032306 conn(0x7fe54c0fd140 msgr2=0x7fe54c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.036+0000 7fe553f77640 1 -- 172.21.15.106:0/4132032306 shutdown_connections 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.036+0000 7fe553f77640 1 -- 172.21.15.106:0/4132032306 wait complete. 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe553f77640 1 Processor -- start 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe553f77640 1 -- start start 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe553f77640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe54c101630 0x7fe54c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe553f77640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101ff0 0x7fe54c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe553f77640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe54c107640 0x7fe54c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe553f77640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c111a50 con 0x7fe54c101ff0 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe553f77640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c1118d0 con 0x7fe54c107640 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe553f77640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c111bd0 con 0x7fe54c101630 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe551512640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101ff0 0x7fe54c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe551512640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101ff0 0x7fe54c198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42726/0 (socket says 172.21.15.106:42726) 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe551512640 1 -- 172.21.15.106:0/1730294089 learned_addr learned my addr 172.21.15.106:0/1730294089 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe551d13640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe54c101630 0x7fe54c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe552514640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe54c107640 0x7fe54c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.037+0000 7fe551512640 1 -- 172.21.15.106:0/1730294089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe54c101630 msgr2=0x7fe54c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe551512640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe54c101630 0x7fe54c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe551512640 1 -- 172.21.15.106:0/1730294089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe54c107640 msgr2=0x7fe54c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe551512640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe54c107640 0x7fe54c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe551512640 1 -- 172.21.15.106:0/1730294089 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c19cfb0 con 0x7fe54c101ff0 2024-03-28T02:21:57.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe552514640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe54c107640 0x7fe54c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:21:57.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe551512640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101ff0 0x7fe54c198500 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe548009230 tx=0x7fe54802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:57.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe542ffd640 1 -- 172.21.15.106:0/1730294089 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548041070 con 0x7fe54c101ff0 2024-03-28T02:21:57.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe542ffd640 1 -- 172.21.15.106:0/1730294089 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe548004450 con 0x7fe54c101ff0 2024-03-28T02:21:57.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe542ffd640 1 -- 172.21.15.106:0/1730294089 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548003de0 con 0x7fe54c101ff0 2024-03-28T02:21:57.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c19d240 con 0x7fe54c101ff0 2024-03-28T02:21:57.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.038+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c1a4b30 con 0x7fe54c101ff0 2024-03-28T02:21:57.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.039+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c0688e0 con 0x7fe54c101ff0 2024-03-28T02:21:57.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.040+0000 7fe542ffd640 1 -- 172.21.15.106:0/1730294089 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe548044070 con 0x7fe54c101ff0 2024-03-28T02:21:57.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.041+0000 7fe542ffd640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe5380a29f0 0x7fe5380a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.041+0000 7fe551d13640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe5380a29f0 0x7fe5380a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.041+0000 7fe542ffd640 1 -- 172.21.15.106:0/1730294089 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7fe5480f09e0 con 0x7fe54c101ff0 2024-03-28T02:21:57.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.041+0000 7fe551d13640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe5380a29f0 0x7fe5380a4eb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009b50 tx=0x7fe53c005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:57.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.045+0000 7fe542ffd640 1 -- 172.21.15.106:0/1730294089 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe5480bd480 con 0x7fe54c101ff0 2024-03-28T02:21:57.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.221+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe54c109610 con 0x7fe54c101ff0 2024-03-28T02:21:57.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.223+0000 7fe542ffd640 1 -- 172.21.15.106:0/1730294089 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+25414 (secure 0 0 0) 0x7fe5480bcd80 con 0x7fe54c101ff0 2024-03-28T02:21:57.216 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:21:57.216 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":102,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:21:13.433026+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6801/3501119323":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/2601893696":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/3655727768":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/899620795":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/3501119323":"2024-03-29T02:11:16.390724+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2664246824":"2024-03-29T02:11:16.390724+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:21:57.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe5380a29f0 msgr2=0x7fe5380a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe5380a29f0 0x7fe5380a4eb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009b50 tx=0x7fe53c005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:57.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101ff0 msgr2=0x7fe54c198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101ff0 0x7fe54c198500 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe548009230 tx=0x7fe54802cb30 comp rx=0 tx=0).stop 2024-03-28T02:21:57.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 shutdown_connections 2024-03-28T02:21:57.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fe5380a29f0 0x7fe5380a4eb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe54c107640 0x7fe54c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe54c101ff0 0x7fe54c198500 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 --2- 172.21.15.106:0/1730294089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe54c101630 0x7fe54c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 >> 172.21.15.106:0/1730294089 conn(0x7fe54c0fd140 msgr2=0x7fe54c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:57.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 shutdown_connections 2024-03-28T02:21:57.219 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.225+0000 7fe553f77640 1 -- 172.21.15.106:0/1730294089 wait complete. 2024-03-28T02:21:57.229 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T02:04:06.605353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T02:08:46.952662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T02:08:47.976273+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T02:08:51.703592+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T02:21:57.230 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T02:21:57.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.342+0000 7f11e384b640 1 -- 172.21.15.106:0/4179077596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11dc107640 msgr2=0x7f11dc107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.342+0000 7f11e384b640 1 --2- 172.21.15.106:0/4179077596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11dc107640 0x7f11dc107a20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f11d8009860 tx=0x7f11d802bb70 comp rx=0 tx=0).stop 2024-03-28T02:21:57.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.342+0000 7f11e384b640 1 -- 172.21.15.106:0/4179077596 shutdown_connections 2024-03-28T02:21:57.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.342+0000 7f11e384b640 1 --2- 172.21.15.106:0/4179077596 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11dc101ff0 0x7f11dc10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.342+0000 7f11e384b640 1 --2- 172.21.15.106:0/4179077596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11dc101630 0x7f11dc101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.342+0000 7f11e384b640 1 --2- 172.21.15.106:0/4179077596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11dc107640 0x7f11dc107a20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.342+0000 7f11e384b640 1 -- 172.21.15.106:0/4179077596 >> 172.21.15.106:0/4179077596 conn(0x7f11dc078aa0 msgr2=0x7f11dc078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:57.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.342+0000 7f11e384b640 1 -- 172.21.15.106:0/4179077596 shutdown_connections 2024-03-28T02:21:57.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.342+0000 7f11e384b640 1 -- 172.21.15.106:0/4179077596 wait complete. 2024-03-28T02:21:57.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e384b640 1 Processor -- start 2024-03-28T02:21:57.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e384b640 1 -- start start 2024-03-28T02:21:57.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e384b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11dc101630 0x7f11dc0739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e384b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11dc101ff0 0x7f11dc073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e384b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11dc107640 0x7f11dc074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e384b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc111a50 con 0x7f11dc107640 2024-03-28T02:21:57.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e384b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc1118d0 con 0x7f11dc101ff0 2024-03-28T02:21:57.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e384b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc111bd0 con 0x7f11dc101630 2024-03-28T02:21:57.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e1de8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11dc107640 0x7f11dc074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e1de8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11dc107640 0x7f11dc074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42738/0 (socket says 172.21.15.106:42738) 2024-03-28T02:21:57.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e1de8640 1 -- 172.21.15.106:0/722329830 learned_addr learned my addr 172.21.15.106:0/722329830 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:57.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e0de6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11dc101ff0 0x7f11dc073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e15e7640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11dc101630 0x7f11dc0739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e1de8640 1 -- 172.21.15.106:0/722329830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11dc101630 msgr2=0x7f11dc0739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e1de8640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11dc101630 0x7f11dc0739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e1de8640 1 -- 172.21.15.106:0/722329830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11dc101ff0 msgr2=0x7f11dc073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e1de8640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11dc101ff0 0x7f11dc073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e1de8640 1 -- 172.21.15.106:0/722329830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc1a6470 con 0x7f11dc107640 2024-03-28T02:21:57.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.343+0000 7f11e1de8640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11dc107640 0x7f11dc074420 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f11d400efc0 tx=0x7f11d400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:57.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.344+0000 7f11d27fc640 1 -- 172.21.15.106:0/722329830 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4003970 con 0x7f11dc107640 2024-03-28T02:21:57.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.344+0000 7f11d27fc640 1 -- 172.21.15.106:0/722329830 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11d4004020 con 0x7f11dc107640 2024-03-28T02:21:57.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.344+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc1a6760 con 0x7f11dc107640 2024-03-28T02:21:57.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.344+0000 7f11d27fc640 1 -- 172.21.15.106:0/722329830 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d400b9f0 con 0x7f11dc107640 2024-03-28T02:21:57.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.344+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc1a6c60 con 0x7f11dc107640 2024-03-28T02:21:57.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.345+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc0688e0 con 0x7f11dc107640 2024-03-28T02:21:57.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.346+0000 7f11d27fc640 1 -- 172.21.15.106:0/722329830 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f11d401d020 con 0x7f11dc107640 2024-03-28T02:21:57.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.346+0000 7f11d27fc640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f11c80a2970 0x7f11c80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.346+0000 7f11e15e7640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f11c80a2970 0x7f11c80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.346+0000 7f11d27fc640 1 -- 172.21.15.106:0/722329830 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f11d40cf640 con 0x7f11dc107640 2024-03-28T02:21:57.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.347+0000 7f11e15e7640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f11c80a2970 0x7f11c80a4e30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f11d802c180 tx=0x7f11d80058d0 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:57.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.350+0000 7f11d27fc640 1 -- 172.21.15.106:0/722329830 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f11d409bfd0 con 0x7f11dc107640 2024-03-28T02:21:57.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.529+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f11dc02d070 con 0x7f11dc107640 2024-03-28T02:21:57.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.529+0000 7f11d27fc640 1 -- 172.21.15.106:0/722329830 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f11d409b8d0 con 0x7f11dc107640 2024-03-28T02:21:57.522 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-28T02:21:57.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.531+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f11c80a2970 msgr2=0x7f11c80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.531+0000 7f11e384b640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f11c80a2970 0x7f11c80a4e30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f11d802c180 tx=0x7f11d80058d0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.531+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11dc107640 msgr2=0x7f11dc074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.531+0000 7f11e384b640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11dc107640 0x7f11dc074420 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f11d400efc0 tx=0x7f11d400c800 comp rx=0 tx=0).stop 2024-03-28T02:21:57.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.532+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 shutdown_connections 2024-03-28T02:21:57.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.532+0000 7f11e384b640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f11c80a2970 0x7f11c80a4e30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.532+0000 7f11e384b640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11dc107640 0x7f11dc074420 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.532+0000 7f11e384b640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11dc101ff0 0x7f11dc073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.532+0000 7f11e384b640 1 --2- 172.21.15.106:0/722329830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11dc101630 0x7f11dc0739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.532+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 >> 172.21.15.106:0/722329830 conn(0x7f11dc078aa0 msgr2=0x7f11dc104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:57.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.532+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 shutdown_connections 2024-03-28T02:21:57.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.532+0000 7f11e384b640 1 -- 172.21.15.106:0/722329830 wait complete. 2024-03-28T02:21:57.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T02:21:57.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 -- 172.21.15.106:0/1147367305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0901005c0 msgr2=0x7fb090100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 --2- 172.21.15.106:0/1147367305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0901005c0 0x7fb090100a40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb080009900 tx=0x7fb08002bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:57.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 -- 172.21.15.106:0/1147367305 shutdown_connections 2024-03-28T02:21:57.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 --2- 172.21.15.106:0/1147367305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb090100f80 0x7fb09010d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 --2- 172.21.15.106:0/1147367305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0901005c0 0x7fb090100a40 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 --2- 172.21.15.106:0/1147367305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb090106630 0x7fb090106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 -- 172.21.15.106:0/1147367305 >> 172.21.15.106:0/1147367305 conn(0x7fb090077700 msgr2=0x7fb090077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 -- 172.21.15.106:0/1147367305 shutdown_connections 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 -- 172.21.15.106:0/1147367305 wait complete. 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.650+0000 7fb095ec0640 1 Processor -- start 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb095ec0640 1 -- start start 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb095ec0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0901005c0 0x7fb0901021e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb095ec0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb090100f80 0x7fb090102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb095ec0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb090106630 0x7fb090102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb095ec0640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb090110910 con 0x7fb090100f80 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb095ec0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb090110790 con 0x7fb0901005c0 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb095ec0640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb090110a90 con 0x7fb090106630 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb090100f80 0x7fb090102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0901005c0 0x7fb0901021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb090100f80 0x7fb090102720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41490/0 (socket says 172.21.15.106:41490) 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb090106630 0x7fb090102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0901005c0 0x7fb0901021e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.106:56726/0 (socket says 172.21.15.106:56726) 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08effd640 1 -- 172.21.15.106:0/3563451294 learned_addr learned my addr 172.21.15.106:0/3563451294 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08effd640 1 -- 172.21.15.106:0/3563451294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb090106630 msgr2=0x7fb090102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08effd640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb090106630 0x7fb090102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08effd640 1 -- 172.21.15.106:0/3563451294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0901005c0 msgr2=0x7fb0901021e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08effd640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0901005c0 0x7fb0901021e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08effd640 1 -- 172.21.15.106:0/3563451294 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0901a6360 con 0x7fb090100f80 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08ffff640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb090106630 0x7fb090102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08f7fe640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0901005c0 0x7fb0901021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:21:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08effd640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb090100f80 0x7fb090102720 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb080005940 tx=0x7fb080004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:57.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.651+0000 7fb08cff9640 1 -- 172.21.15.106:0/3563451294 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080041070 con 0x7fb090100f80 2024-03-28T02:21:57.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.652+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0901a65f0 con 0x7fb090100f80 2024-03-28T02:21:57.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.652+0000 7fb08cff9640 1 -- 172.21.15.106:0/3563451294 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0800054e0 con 0x7fb090100f80 2024-03-28T02:21:57.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.652+0000 7fb08cff9640 1 -- 172.21.15.106:0/3563451294 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080003d00 con 0x7fb090100f80 2024-03-28T02:21:57.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.652+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0901a6b30 con 0x7fb090100f80 2024-03-28T02:21:57.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.652+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb090067c00 con 0x7fb090100f80 2024-03-28T02:21:57.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.654+0000 7fb08cff9640 1 -- 172.21.15.106:0/3563451294 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fb080044070 con 0x7fb090100f80 2024-03-28T02:21:57.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.654+0000 7fb08cff9640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb0780a2970 0x7fb0780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.654+0000 7fb08f7fe640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb0780a2970 0x7fb0780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.654+0000 7fb08cff9640 1 -- 172.21.15.106:0/3563451294 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7fb0800f1000 con 0x7fb090100f80 2024-03-28T02:21:57.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.655+0000 7fb08f7fe640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb0780a2970 0x7fb0780a4e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb084009f40 tx=0x7fb084005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:57.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.658+0000 7fb08cff9640 1 -- 172.21.15.106:0/3563451294 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb0800bd450 con 0x7fb090100f80 2024-03-28T02:21:57.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.831+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0901a6e50 con 0x7fb090100f80 2024-03-28T02:21:57.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.831+0000 7fb08cff9640 1 -- 172.21.15.106:0/3563451294 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0800c14f0 con 0x7fb090100f80 2024-03-28T02:21:57.825 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb0780a2970 msgr2=0x7fb0780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb0780a2970 0x7fb0780a4e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb084009f40 tx=0x7fb084005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb090100f80 msgr2=0x7fb090102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb090100f80 0x7fb090102720 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb080005940 tx=0x7fb080004270 comp rx=0 tx=0).stop 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 shutdown_connections 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fb0780a2970 0x7fb0780a4e30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb090106630 0x7fb090102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb090100f80 0x7fb090102720 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 --2- 172.21.15.106:0/3563451294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0901005c0 0x7fb0901021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 >> 172.21.15.106:0/3563451294 conn(0x7fb090077700 msgr2=0x7fb0900ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 shutdown_connections 2024-03-28T02:21:57.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.833+0000 7fb095ec0640 1 -- 172.21.15.106:0/3563451294 wait complete. 2024-03-28T02:21:57.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:57 smithi106 ceph-mon[31965]: pgmap v331: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:57.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:57 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1730294089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:21:57.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:57 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/722329830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:21:57.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T02:21:57.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:57 smithi178 ceph-mon[36521]: pgmap v331: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:57.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:57 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1730294089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:21:57.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:57 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/722329830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.949+0000 7efe025f4640 1 -- 172.21.15.106:0/2869550425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100530 msgr2=0x7efdfc1009b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.949+0000 7efe025f4640 1 --2- 172.21.15.106:0/2869550425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100530 0x7efdfc1009b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009900 tx=0x7efdf002bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.949+0000 7efe025f4640 1 -- 172.21.15.106:0/2869550425 shutdown_connections 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.949+0000 7efe025f4640 1 --2- 172.21.15.106:0/2869550425 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc100ef0 0x7efdfc10d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.949+0000 7efe025f4640 1 --2- 172.21.15.106:0/2869550425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100530 0x7efdfc1009b0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.949+0000 7efe025f4640 1 --2- 172.21.15.106:0/2869550425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdfc1065f0 0x7efdfc1069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.949+0000 7efe025f4640 1 -- 172.21.15.106:0/2869550425 >> 172.21.15.106:0/2869550425 conn(0x7efdfc0776a0 msgr2=0x7efdfc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.949+0000 7efe025f4640 1 -- 172.21.15.106:0/2869550425 shutdown_connections 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.949+0000 7efe025f4640 1 -- 172.21.15.106:0/2869550425 wait complete. 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efe025f4640 1 Processor -- start 2024-03-28T02:21:57.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efe025f4640 1 -- start start 2024-03-28T02:21:57.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efe025f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdfc100530 0x7efdfc197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efe025f4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100ef0 0x7efdfc1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efe025f4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc1065f0 0x7efdfc19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efe025f4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1108d0 con 0x7efdfc100ef0 2024-03-28T02:21:57.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efe025f4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc110750 con 0x7efdfc100530 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efe025f4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc110a50 con 0x7efdfc1065f0 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100ef0 0x7efdfc1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100ef0 0x7efdfc1983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41496/0 (socket says 172.21.15.106:41496) 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdfc100530 0x7efdfc197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfb7fe640 1 -- 172.21.15.106:0/4135643093 learned_addr learned my addr 172.21.15.106:0/4135643093 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efe00b91640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc1065f0 0x7efdfc19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfb7fe640 1 -- 172.21.15.106:0/4135643093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc1065f0 msgr2=0x7efdfc19c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfb7fe640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc1065f0 0x7efdfc19c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfb7fe640 1 -- 172.21.15.106:0/4135643093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdfc100530 msgr2=0x7efdfc197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfb7fe640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdfc100530 0x7efdfc197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfb7fe640 1 -- 172.21.15.106:0/4135643093 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc19ce20 con 0x7efdfc100ef0 2024-03-28T02:21:57.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.950+0000 7efdfbfff640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdfc100530 0x7efdfc197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:21:57.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.951+0000 7efdfb7fe640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100ef0 0x7efdfc1983c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005940 tx=0x7efdf0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:57.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.951+0000 7efdf97fa640 1 -- 172.21.15.106:0/4135643093 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0041070 con 0x7efdfc100ef0 2024-03-28T02:21:57.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.951+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc19d0b0 con 0x7efdfc100ef0 2024-03-28T02:21:57.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.951+0000 7efdf97fa640 1 -- 172.21.15.106:0/4135643093 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf00054e0 con 0x7efdfc100ef0 2024-03-28T02:21:57.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.951+0000 7efdf97fa640 1 -- 172.21.15.106:0/4135643093 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0003d00 con 0x7efdfc100ef0 2024-03-28T02:21:57.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.951+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc1a49a0 con 0x7efdfc100ef0 2024-03-28T02:21:57.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.952+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdfc068490 con 0x7efdfc100ef0 2024-03-28T02:21:57.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.953+0000 7efdf97fa640 1 -- 172.21.15.106:0/4135643093 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7efdf0044070 con 0x7efdfc100ef0 2024-03-28T02:21:57.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.954+0000 7efdf97fa640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7efde40a2970 0x7efde40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:57.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.954+0000 7efdfbfff640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7efde40a2970 0x7efde40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:57.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.954+0000 7efdf97fa640 1 -- 172.21.15.106:0/4135643093 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7efdf00f0fd0 con 0x7efdfc100ef0 2024-03-28T02:21:57.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.954+0000 7efdfbfff640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7efde40a2970 0x7efde40a4e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efdec0095f0 tx=0x7efdec005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:57.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:57.957+0000 7efdf97fa640 1 -- 172.21.15.106:0/4135643093 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efdf00bd3e0 con 0x7efdfc100ef0 2024-03-28T02:21:57.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:57 smithi153 ceph-mon[36228]: pgmap v331: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T02:21:57.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:57 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1730294089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:21:57.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:57 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/722329830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T02:21:58.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.131+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efdfc1a4bd0 con 0x7efdfc100ef0 2024-03-28T02:21:58.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.132+0000 7efdf97fa640 1 -- 172.21.15.106:0/4135643093 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7efdf00c1480 con 0x7efdfc100ef0 2024-03-28T02:21:58.125 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 64 2024-03-28T02:21:58.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.133+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7efde40a2970 msgr2=0x7efde40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.133+0000 7efe025f4640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7efde40a2970 0x7efde40a4e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efdec0095f0 tx=0x7efdec005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:58.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.133+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100ef0 msgr2=0x7efdfc1983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.133+0000 7efe025f4640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100ef0 0x7efdfc1983c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005940 tx=0x7efdf0004270 comp rx=0 tx=0).stop 2024-03-28T02:21:58.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.134+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 shutdown_connections 2024-03-28T02:21:58.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.134+0000 7efe025f4640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7efde40a2970 0x7efde40a4e30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.134+0000 7efe025f4640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc1065f0 0x7efdfc19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.134+0000 7efe025f4640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdfc100ef0 0x7efdfc1983c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.134+0000 7efe025f4640 1 --2- 172.21.15.106:0/4135643093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdfc100530 0x7efdfc197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.134+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 >> 172.21.15.106:0/4135643093 conn(0x7efdfc0776a0 msgr2=0x7efdfc0689e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:58.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.134+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 shutdown_connections 2024-03-28T02:21:58.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.134+0000 7efe025f4640 1 -- 172.21.15.106:0/4135643093 wait complete. 2024-03-28T02:21:58.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T02:21:58.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 -- 172.21.15.106:0/3409796126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa74106630 msgr2=0x7faa74106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/3409796126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa74106630 0x7faa74106a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faa70009890 tx=0x7faa7002bb70 comp rx=0 tx=0).stop 2024-03-28T02:21:58.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 -- 172.21.15.106:0/3409796126 shutdown_connections 2024-03-28T02:21:58.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/3409796126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa74100f30 0x7faa7410d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/3409796126 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa74100570 0x7faa741009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/3409796126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa74106630 0x7faa74106a10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 -- 172.21.15.106:0/3409796126 >> 172.21.15.106:0/3409796126 conn(0x7faa74076920 msgr2=0x7faa74076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 -- 172.21.15.106:0/3409796126 shutdown_connections 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 -- 172.21.15.106:0/3409796126 wait complete. 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 Processor -- start 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.249+0000 7faa7c7bc640 1 -- start start 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7c7bc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa74100570 0x7faa74197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7c7bc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa74100f30 0x7faa74198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7c7bc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa74106630 0x7faa7419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7c7bc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa741108f0 con 0x7faa74100570 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7c7bc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa74110770 con 0x7faa74106630 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7c7bc640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa74110a70 con 0x7faa74100f30 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7a558640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa74100570 0x7faa74197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7a558640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa74100570 0x7faa74197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41520/0 (socket says 172.21.15.106:41520) 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7a558640 1 -- 172.21.15.106:0/985022731 learned_addr learned my addr 172.21.15.106:0/985022731 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa79d57640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa74100f30 0x7faa74198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7ad59640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa74106630 0x7faa7419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:58.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7a558640 1 -- 172.21.15.106:0/985022731 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa74100f30 msgr2=0x7faa74198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7a558640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa74100f30 0x7faa74198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7a558640 1 -- 172.21.15.106:0/985022731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa74106630 msgr2=0x7faa7419c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7a558640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa74106630 0x7faa7419c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7a558640 1 -- 172.21.15.106:0/985022731 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7419ce70 con 0x7faa74100570 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7a558640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa74100570 0x7faa74197ed0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faa70000c00 tx=0x7faa70004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa6b7fe640 1 -- 172.21.15.106:0/985022731 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70041020 con 0x7faa74100570 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7419d100 con 0x7faa74100570 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa6b7fe640 1 -- 172.21.15.106:0/985022731 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa7002c9f0 con 0x7faa74100570 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa6b7fe640 1 -- 172.21.15.106:0/985022731 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70039040 con 0x7faa74100570 2024-03-28T02:21:58.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.250+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa741a49f0 con 0x7faa74100570 2024-03-28T02:21:58.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.251+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa74101c90 con 0x7faa74100570 2024-03-28T02:21:58.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.252+0000 7faa6b7fe640 1 -- 172.21.15.106:0/985022731 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7faa70044020 con 0x7faa74100570 2024-03-28T02:21:58.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.253+0000 7faa6b7fe640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7faa600a2970 0x7faa600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:58.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.253+0000 7faa79d57640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7faa600a2970 0x7faa600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:58.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.253+0000 7faa6b7fe640 1 -- 172.21.15.106:0/985022731 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7faa700f0920 con 0x7faa74100570 2024-03-28T02:21:58.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.253+0000 7faa79d57640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7faa600a2970 0x7faa600a4e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faa74105c30 tx=0x7faa64009260 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:58.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.257+0000 7faa6b7fe640 1 -- 172.21.15.106:0/985022731 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faa7004b020 con 0x7faa74100570 2024-03-28T02:21:58.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.432+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7faa741a4bc0 con 0x7faa74100570 2024-03-28T02:21:58.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.433+0000 7faa6b7fe640 1 -- 172.21.15.106:0/985022731 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) v1 ==== 103+0+11 (secure 0 0 0) 0x7faa700bd3c0 con 0x7faa74100570 2024-03-28T02:21:58.426 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 32 2024-03-28T02:21:58.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7faa600a2970 msgr2=0x7faa600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7faa600a2970 0x7faa600a4e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faa74105c30 tx=0x7faa64009260 comp rx=0 tx=0).stop 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa74100570 msgr2=0x7faa74197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa74100570 0x7faa74197ed0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faa70000c00 tx=0x7faa70004550 comp rx=0 tx=0).stop 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 shutdown_connections 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7faa600a2970 0x7faa600a4e30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa74106630 0x7faa7419c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa74100f30 0x7faa74198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 --2- 172.21.15.106:0/985022731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa74100570 0x7faa74197ed0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 >> 172.21.15.106:0/985022731 conn(0x7faa74076920 msgr2=0x7faa740ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 shutdown_connections 2024-03-28T02:21:58.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.435+0000 7faa7c7bc640 1 -- 172.21.15.106:0/985022731 wait complete. 2024-03-28T02:21:58.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T02:21:58.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 -- 172.21.15.106:0/2295882991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 msgr2=0x7f3f3410e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 --2- 172.21.15.106:0/2295882991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 0x7f3f3410e500 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009cd0 tx=0x7f3f2c02c0f0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 -- 172.21.15.106:0/2295882991 shutdown_connections 2024-03-28T02:21:58.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 --2- 172.21.15.106:0/2295882991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 0x7f3f3410e500 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 --2- 172.21.15.106:0/2295882991 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f34101630 0x7f3f34101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 --2- 172.21.15.106:0/2295882991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f34107640 0x7f3f34107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 -- 172.21.15.106:0/2295882991 >> 172.21.15.106:0/2295882991 conn(0x7f3f34078aa0 msgr2=0x7f3f34078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 -- 172.21.15.106:0/2295882991 shutdown_connections 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 -- 172.21.15.106:0/2295882991 wait complete. 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 Processor -- start 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.549+0000 7f3f39a75640 1 -- start start 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f39a75640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f34101630 0x7f3f34197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f39a75640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 0x7f3f34198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f39a75640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f34107640 0x7f3f3419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f39a75640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f34111a50 con 0x7f3f34101ff0 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f39a75640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f341118d0 con 0x7f3f34107640 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f39a75640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f34111bd0 con 0x7f3f34101630 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f2affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 0x7f3f34198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f2affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 0x7f3f34198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41532/0 (socket says 172.21.15.106:41532) 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f2affd640 1 -- 172.21.15.106:0/4015441394 learned_addr learned my addr 172.21.15.106:0/4015441394 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f337fe640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f34101630 0x7f3f34197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f33fff640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f34107640 0x7f3f3419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f2affd640 1 -- 172.21.15.106:0/4015441394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f34101630 msgr2=0x7f3f34197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f2affd640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f34101630 0x7f3f34197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f2affd640 1 -- 172.21.15.106:0/4015441394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f34107640 msgr2=0x7f3f3419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f2affd640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f34107640 0x7f3f3419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f2affd640 1 -- 172.21.15.106:0/4015441394 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3419cef0 con 0x7f3f34101ff0 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f2affd640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 0x7f3f34198490 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00ea60 tx=0x7f3f1c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f317fa640 1 -- 172.21.15.106:0/4015441394 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c00cde0 con 0x7f3f34101ff0 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f317fa640 1 -- 172.21.15.106:0/4015441394 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f1c0035f0 con 0x7f3f34101ff0 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3419d1e0 con 0x7f3f34101ff0 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f317fa640 1 -- 172.21.15.106:0/4015441394 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c010670 con 0x7f3f34101ff0 2024-03-28T02:21:58.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.550+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f341a4a70 con 0x7f3f34101ff0 2024-03-28T02:21:58.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.551+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f340688e0 con 0x7f3f34101ff0 2024-03-28T02:21:58.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.553+0000 7f3f317fa640 1 -- 172.21.15.106:0/4015441394 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f3f1c0037a0 con 0x7f3f34101ff0 2024-03-28T02:21:58.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.553+0000 7f3f317fa640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f140a2970 0x7f3f140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:58.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.553+0000 7f3f337fe640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f140a2970 0x7f3f140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:58.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.553+0000 7f3f317fa640 1 -- 172.21.15.106:0/4015441394 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f3f1c0cf520 con 0x7f3f34101ff0 2024-03-28T02:21:58.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.554+0000 7f3f337fe640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f140a2970 0x7f3f140a4e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009b50 tx=0x7f3f24005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:58.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.558+0000 7f3f317fa640 1 -- 172.21.15.106:0/4015441394 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f1c09beb0 con 0x7f3f34101ff0 2024-03-28T02:21:58.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.748+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3f3402d070 con 0x7f3f34101ff0 2024-03-28T02:21:58.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.748+0000 7f3f317fa640 1 -- 172.21.15.106:0/4015441394 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v102) v1 ==== 81+0+955 (secure 0 0 0) 0x7f3f1c09b7b0 con 0x7f3f34101ff0 2024-03-28T02:21:58.742 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6801/3501119323 2024-03-29T02:11:16.390724+0000 2024-03-28T02:21:58.742 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/2601893696 2024-03-29T02:11:16.390724+0000 2024-03-28T02:21:58.742 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/3655727768 2024-03-29T02:11:16.390724+0000 2024-03-28T02:21:58.742 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/11266040 2024-03-29T02:01:42.267941+0000 2024-03-28T02:21:58.742 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/2884389364 2024-03-29T02:01:42.267941+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/899620795 2024-03-29T02:11:16.390724+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/4050392782 2024-03-29T02:01:42.267941+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6800/3501119323 2024-03-29T02:11:16.390724+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6800/1510653090 2024-03-29T02:01:15.853032+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/103827077 2024-03-29T02:01:15.853032+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/2664246824 2024-03-29T02:11:16.390724+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6801/305853075 2024-03-29T02:01:42.267941+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/256234961 2024-03-29T02:01:15.853032+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6800/305853075 2024-03-29T02:01:42.267941+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:6801/1510653090 2024-03-29T02:01:15.853032+0000 2024-03-28T02:21:58.743 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106:0/2759478196 2024-03-29T02:01:15.853032+0000 2024-03-28T02:21:58.744 INFO:teuthology.orchestra.run.smithi106.stderr:listed 16 entries 2024-03-28T02:21:58.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f140a2970 msgr2=0x7f3f140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f140a2970 0x7f3f140a4e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009b50 tx=0x7f3f24005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:58.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 msgr2=0x7f3f34198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:58.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 0x7f3f34198490 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00ea60 tx=0x7f3f1c00ef20 comp rx=0 tx=0).stop 2024-03-28T02:21:58.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 shutdown_connections 2024-03-28T02:21:58.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f3f140a2970 0x7f3f140a4e30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f34107640 0x7f3f3419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f34101ff0 0x7f3f34198490 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 --2- 172.21.15.106:0/4015441394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f34101630 0x7f3f34197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:58.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 >> 172.21.15.106:0/4015441394 conn(0x7f3f34078aa0 msgr2=0x7f3f34104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:58.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 shutdown_connections 2024-03-28T02:21:58.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:58.750+0000 7f3f39a75640 1 -- 172.21.15.106:0/4015441394 wait complete. 2024-03-28T02:21:58.753 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:58 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3563451294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:21:58.753 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:58 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4135643093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:21:58.753 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:58 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/985022731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:21:58.754 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-28T02:21:58.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:21:58.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-28T02:21:58.799 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:21:58.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:21:58.800 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T02:21:58.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:21:58.822 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list 2024-03-28T02:21:58.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:58 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3563451294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:21:58.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:58 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4135643093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:21:58.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:58 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/985022731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:21:58.878 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T02:21:58.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:21:58.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T02:21:58.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T02:21:58.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link delete ceph-brx 2024-03-28T02:21:58.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:58 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3563451294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T02:21:58.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:58 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4135643093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T02:21:58.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:58 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/985022731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T02:21:59.044 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-28T02:21:59.054 INFO:tasks.cephadm:Teardown begin 2024-03-28T02:21:59.054 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T02:21:59.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:21:59.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.166+0000 7f08f03eb640 1 -- 172.21.15.106:0/2437136406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e806a6f0 msgr2=0x7f08e810e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.166+0000 7f08f03eb640 1 --2- 172.21.15.106:0/2437136406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e806a6f0 0x7f08e810e4f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009900 tx=0x7f08e402bcd0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.166+0000 7f08f03eb640 1 -- 172.21.15.106:0/2437136406 shutdown_connections 2024-03-28T02:21:59.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.166+0000 7f08f03eb640 1 --2- 172.21.15.106:0/2437136406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e806a6f0 0x7f08e810e4f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.166+0000 7f08f03eb640 1 --2- 172.21.15.106:0/2437136406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e8069d50 0x7f08e806a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.166+0000 7f08f03eb640 1 --2- 172.21.15.106:0/2437136406 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08e81076a0 0x7f08e8107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.166+0000 7f08f03eb640 1 -- 172.21.15.106:0/2437136406 >> 172.21.15.106:0/2437136406 conn(0x7f08e8077640 msgr2=0x7f08e8077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:59.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.166+0000 7f08f03eb640 1 -- 172.21.15.106:0/2437136406 shutdown_connections 2024-03-28T02:21:59.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.166+0000 7f08f03eb640 1 -- 172.21.15.106:0/2437136406 wait complete. 2024-03-28T02:21:59.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08f03eb640 1 Processor -- start 2024-03-28T02:21:59.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08f03eb640 1 -- start start 2024-03-28T02:21:59.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08f03eb640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e8069d50 0x7f08e81005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08f03eb640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08e806a6f0 0x7f08e8100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08f03eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e81076a0 0x7f08e8101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08f03eb640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8111a60 con 0x7f08e8069d50 2024-03-28T02:21:59.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08f03eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e81118e0 con 0x7f08e81076a0 2024-03-28T02:21:59.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08f03eb640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8111be0 con 0x7f08e806a6f0 2024-03-28T02:21:59.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee187640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e8069d50 0x7f08e81005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee187640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e8069d50 0x7f08e81005d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41548/0 (socket says 172.21.15.106:41548) 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee187640 1 -- 172.21.15.106:0/3673057762 learned_addr learned my addr 172.21.15.106:0/3673057762 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ed986640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08e806a6f0 0x7f08e8100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee988640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e81076a0 0x7f08e8101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee187640 1 -- 172.21.15.106:0/3673057762 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08e806a6f0 msgr2=0x7f08e8100b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee187640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08e806a6f0 0x7f08e8100b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee187640 1 -- 172.21.15.106:0/3673057762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e81076a0 msgr2=0x7f08e8101050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee187640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e81076a0 0x7f08e8101050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee187640 1 -- 172.21.15.106:0/3673057762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e81a63c0 con 0x7f08e8069d50 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee988640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e81076a0 0x7f08e8101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08ee187640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e8069d50 0x7f08e81005d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f08e000eb90 tx=0x7f08e000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08df7fe640 1 -- 172.21.15.106:0/3673057762 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0004d20 con 0x7f08e8069d50 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08df7fe640 1 -- 172.21.15.106:0/3673057762 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08e000ee90 con 0x7f08e8069d50 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e81a66b0 con 0x7f08e8069d50 2024-03-28T02:21:59.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.167+0000 7f08df7fe640 1 -- 172.21.15.106:0/3673057762 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0010830 con 0x7f08e8069d50 2024-03-28T02:21:59.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.168+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e81a6bb0 con 0x7f08e8069d50 2024-03-28T02:21:59.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.168+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e8102dc0 con 0x7f08e8069d50 2024-03-28T02:21:59.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.170+0000 7f08df7fe640 1 -- 172.21.15.106:0/3673057762 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f08e0005000 con 0x7f08e8069d50 2024-03-28T02:21:59.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.170+0000 7f08df7fe640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f08d40a2970 0x7f08d40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.170+0000 7f08ed986640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f08d40a2970 0x7f08d40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.170+0000 7f08df7fe640 1 -- 172.21.15.106:0/3673057762 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f08e00cfab0 con 0x7f08e8069d50 2024-03-28T02:21:59.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.171+0000 7f08ed986640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f08d40a2970 0x7f08d40a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f08e8102060 tx=0x7f08d8005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:59.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.174+0000 7f08df7fe640 1 -- 172.21.15.106:0/3673057762 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f08e0053740 con 0x7f08e8069d50 2024-03-28T02:21:59.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.386+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08e80688e0 con 0x7f08e8069d50 2024-03-28T02:21:59.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.387+0000 7f08df7fe640 1 -- 172.21.15.106:0/3673057762 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11474 (secure 0 0 0) 0x7f08e009c440 con 0x7f08e8069d50 2024-03-28T02:21:59.381 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:21:59.381 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:11:07.761198+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24499,"mds_2":24460,"mds_3":24446,"mds_4":24415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14625":{"gid":14625,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14637":{"gid":14637,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24446":{"gid":24446,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24460,24446,24509,24415,24499]},"id":1}]} 2024-03-28T02:21:59.381 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 29 2024-03-28T02:21:59.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.389+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f08d40a2970 msgr2=0x7f08d40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.389+0000 7f08f03eb640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f08d40a2970 0x7f08d40a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f08e8102060 tx=0x7f08d8005f90 comp rx=0 tx=0).stop 2024-03-28T02:21:59.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.389+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e8069d50 msgr2=0x7f08e81005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.389+0000 7f08f03eb640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e8069d50 0x7f08e81005d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f08e000eb90 tx=0x7f08e000c6a0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.389+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 shutdown_connections 2024-03-28T02:21:59.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.389+0000 7f08f03eb640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f08d40a2970 0x7f08d40a4e30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.389+0000 7f08f03eb640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e81076a0 0x7f08e8101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.389+0000 7f08f03eb640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08e806a6f0 0x7f08e8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.389+0000 7f08f03eb640 1 --2- 172.21.15.106:0/3673057762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e8069d50 0x7f08e81005d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.390+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 >> 172.21.15.106:0/3673057762 conn(0x7f08e8077640 msgr2=0x7f08e80ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:59.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.390+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 shutdown_connections 2024-03-28T02:21:59.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.390+0000 7f08f03eb640 1 -- 172.21.15.106:0/3673057762 wait complete. 2024-03-28T02:21:59.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T02:21:59.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 -- 172.21.15.106:0/3602209964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c107640 msgr2=0x7fca1c107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 --2- 172.21.15.106:0/3602209964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c107640 0x7fca1c107a20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fca04009930 tx=0x7fca0402bc30 comp rx=0 tx=0).stop 2024-03-28T02:21:59.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 -- 172.21.15.106:0/3602209964 shutdown_connections 2024-03-28T02:21:59.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 --2- 172.21.15.106:0/3602209964 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca1c101ff0 0x7fca1c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 --2- 172.21.15.106:0/3602209964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca1c101630 0x7fca1c101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 --2- 172.21.15.106:0/3602209964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c107640 0x7fca1c107a20 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 -- 172.21.15.106:0/3602209964 >> 172.21.15.106:0/3602209964 conn(0x7fca1c0fd140 msgr2=0x7fca1c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:59.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 -- 172.21.15.106:0/3602209964 shutdown_connections 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 -- 172.21.15.106:0/3602209964 wait complete. 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 Processor -- start 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 -- start start 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.503+0000 7fca22bbc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca1c101630 0x7fca1c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca22bbc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c101ff0 0x7fca1c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca22bbc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca1c107640 0x7fca1c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca22bbc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c111a50 con 0x7fca1c101ff0 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca22bbc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c1118d0 con 0x7fca1c101630 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca22bbc640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c111bd0 con 0x7fca1c107640 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca1bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c101ff0 0x7fca1c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca20958640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca1c101630 0x7fca1c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca1bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c101ff0 0x7fca1c1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41562/0 (socket says 172.21.15.106:41562) 2024-03-28T02:21:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca1bfff640 1 -- 172.21.15.106:0/2947837685 learned_addr learned my addr 172.21.15.106:0/2947837685 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca21159640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca1c107640 0x7fca1c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca1bfff640 1 -- 172.21.15.106:0/2947837685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca1c107640 msgr2=0x7fca1c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca1bfff640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca1c107640 0x7fca1c19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca1bfff640 1 -- 172.21.15.106:0/2947837685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca1c101630 msgr2=0x7fca1c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca1bfff640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca1c101630 0x7fca1c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca1bfff640 1 -- 172.21.15.106:0/2947837685 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c19cf80 con 0x7fca1c101ff0 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca20958640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca1c101630 0x7fca1c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca1bfff640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c101ff0 0x7fca1c1984d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00e9f0 tx=0x7fca0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca19ffb640 1 -- 172.21.15.106:0/2947837685 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c00cd80 con 0x7fca1c101ff0 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca19ffb640 1 -- 172.21.15.106:0/2947837685 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca0c010910 con 0x7fca1c101ff0 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca19ffb640 1 -- 172.21.15.106:0/2947837685 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c018990 con 0x7fca1c101ff0 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c19d270 con 0x7fca1c101ff0 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.504+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c1a4b00 con 0x7fca1c101ff0 2024-03-28T02:21:59.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.505+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1c0688e0 con 0x7fca1c101ff0 2024-03-28T02:21:59.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.506+0000 7fca19ffb640 1 -- 172.21.15.106:0/2947837685 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fca0c010ac0 con 0x7fca1c101ff0 2024-03-28T02:21:59.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.507+0000 7fca19ffb640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fca080a2970 0x7fca080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.507+0000 7fca20958640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fca080a2970 0x7fca080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.507+0000 7fca19ffb640 1 -- 172.21.15.106:0/2947837685 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7fca0c0cf530 con 0x7fca1c101ff0 2024-03-28T02:21:59.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.507+0000 7fca20958640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fca080a2970 0x7fca080a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fca04005ab0 tx=0x7fca04005fd0 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:59.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.511+0000 7fca19ffb640 1 -- 172.21.15.106:0/2947837685 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fca0c09b8b0 con 0x7fca1c101ff0 2024-03-28T02:21:59.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.729+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca1c02d070 con 0x7fca1c101ff0 2024-03-28T02:21:59.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.730+0000 7fca19ffb640 1 -- 172.21.15.106:0/2947837685 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11474 (secure 0 0 0) 0x7fca0c09f950 con 0x7fca1c101ff0 2024-03-28T02:21:59.723 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:21:59.723 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14529,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3072695267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3072695267},{"type":"v1","addr":"172.21.15.106:6835","nonce":3072695267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/497398504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":497398504},{"type":"v1","addr":"172.21.15.106:6837","nonce":497398504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T02:08:48.782692+0000","modified":"2024-03-28T02:11:07.761198+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24499,"mds_2":24460,"mds_3":24446,"mds_4":24415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14625":{"gid":14625,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6839/1515774363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":1515774363},{"type":"v1","addr":"172.21.15.106:6839","nonce":1515774363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14637":{"gid":14637,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6841/1766684450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6840","nonce":1766684450},{"type":"v1","addr":"172.21.15.106:6841","nonce":1766684450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/564442660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":564442660},{"type":"v1","addr":"172.21.15.153:6833","nonce":564442660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.178:6833/1496777187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1496777187},{"type":"v1","addr":"172.21.15.178:6833","nonce":1496777187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24446":{"gid":24446,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6835/3469968496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3469968496},{"type":"v1","addr":"172.21.15.153:6835","nonce":3469968496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.178:6835/1402366472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1402366472},{"type":"v1","addr":"172.21.15.178:6835","nonce":1402366472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/3554528514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3554528514},{"type":"v1","addr":"172.21.15.178:6837","nonce":3554528514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/1253460068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1253460068},{"type":"v1","addr":"172.21.15.153:6837","nonce":1253460068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.178:6839/937216083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":937216083},{"type":"v1","addr":"172.21.15.178:6839","nonce":937216083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/505906484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":505906484},{"type":"v1","addr":"172.21.15.153:6839","nonce":505906484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24460,24446,24509,24415,24499]},"id":1}]} 2024-03-28T02:21:59.723 INFO:teuthology.orchestra.run.smithi106.stderr:dumped fsmap epoch 29 2024-03-28T02:21:59.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fca080a2970 msgr2=0x7fca080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fca080a2970 0x7fca080a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fca04005ab0 tx=0x7fca04005fd0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c101ff0 msgr2=0x7fca1c1984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c101ff0 0x7fca1c1984d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00e9f0 tx=0x7fca0c00eeb0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 shutdown_connections 2024-03-28T02:21:59.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fca080a2970 0x7fca080a4e30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca1c107640 0x7fca1c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca1c101ff0 0x7fca1c1984d0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 --2- 172.21.15.106:0/2947837685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca1c101630 0x7fca1c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 >> 172.21.15.106:0/2947837685 conn(0x7fca1c0fd140 msgr2=0x7fca1c0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:59.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 shutdown_connections 2024-03-28T02:21:59.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.732+0000 7fca22bbc640 1 -- 172.21.15.106:0/2947837685 wait complete. 2024-03-28T02:21:59.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:59 smithi106 ceph-mon[31965]: pgmap v332: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:59.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:59 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/4015441394' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T02:21:59.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:21:59 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3673057762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:21:59.736 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T02:21:59.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.844+0000 7f9f7f537640 1 -- 172.21.15.106:0/1875303299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101ff0 msgr2=0x7f9f7810e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.844+0000 7f9f7f537640 1 --2- 172.21.15.106:0/1875303299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101ff0 0x7f9f7810e500 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009cd0 tx=0x7f9f7002c0f0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.844+0000 7f9f7f537640 1 -- 172.21.15.106:0/1875303299 shutdown_connections 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.844+0000 7f9f7f537640 1 --2- 172.21.15.106:0/1875303299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101ff0 0x7f9f7810e500 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.844+0000 7f9f7f537640 1 --2- 172.21.15.106:0/1875303299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f78101630 0x7f9f78101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.844+0000 7f9f7f537640 1 --2- 172.21.15.106:0/1875303299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f78107640 0x7f9f78107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.844+0000 7f9f7f537640 1 -- 172.21.15.106:0/1875303299 >> 172.21.15.106:0/1875303299 conn(0x7f9f780fd140 msgr2=0x7f9f780ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.844+0000 7f9f7f537640 1 -- 172.21.15.106:0/1875303299 shutdown_connections 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.844+0000 7f9f7f537640 1 -- 172.21.15.106:0/1875303299 wait complete. 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7f537640 1 Processor -- start 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7f537640 1 -- start start 2024-03-28T02:21:59.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7f537640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101630 0x7f9f78197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7f537640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f78101ff0 0x7f9f78198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7f537640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f78107640 0x7f9f7819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7f537640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78111a50 con 0x7f9f78101630 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7f537640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f781118d0 con 0x7f9f78107640 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7f537640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78111bd0 con 0x7f9f78101ff0 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7d2d3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101630 0x7f9f78197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7d2d3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101630 0x7f9f78197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41586/0 (socket says 172.21.15.106:41586) 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7d2d3640 1 -- 172.21.15.106:0/2550110560 learned_addr learned my addr 172.21.15.106:0/2550110560 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7cad2640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f78101ff0 0x7f9f78198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7dad4640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f78107640 0x7f9f7819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7d2d3640 1 -- 172.21.15.106:0/2550110560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f78101ff0 msgr2=0x7f9f78198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7d2d3640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f78101ff0 0x7f9f78198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7d2d3640 1 -- 172.21.15.106:0/2550110560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f78107640 msgr2=0x7f9f7819c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:21:59.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7d2d3640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f78107640 0x7f9f7819c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:21:59.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.845+0000 7f9f7d2d3640 1 -- 172.21.15.106:0/2550110560 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7819ced0 con 0x7f9f78101630 2024-03-28T02:21:59.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.846+0000 7f9f7d2d3640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101630 0x7f9f78197f30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9f7400e9f0 tx=0x7f9f7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:59.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.846+0000 7f9f6e7fc640 1 -- 172.21.15.106:0/2550110560 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7400cd80 con 0x7f9f78101630 2024-03-28T02:21:59.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.846+0000 7f9f6e7fc640 1 -- 172.21.15.106:0/2550110560 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f74010910 con 0x7f9f78101630 2024-03-28T02:21:59.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.846+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7819d1c0 con 0x7f9f78101630 2024-03-28T02:21:59.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.846+0000 7f9f6e7fc640 1 -- 172.21.15.106:0/2550110560 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f74018990 con 0x7f9f78101630 2024-03-28T02:21:59.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.846+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f781a4a50 con 0x7f9f78101630 2024-03-28T02:21:59.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.847+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f780688e0 con 0x7f9f78101630 2024-03-28T02:21:59.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.848+0000 7f9f6e7fc640 1 -- 172.21.15.106:0/2550110560 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f9f74010ac0 con 0x7f9f78101630 2024-03-28T02:21:59.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.848+0000 7f9f6e7fc640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f9f640a29f0 0x7f9f640a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:21:59.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.849+0000 7f9f7cad2640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f9f640a29f0 0x7f9f640a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:21:59.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.849+0000 7f9f6e7fc640 1 -- 172.21.15.106:0/2550110560 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7f9f740cf850 con 0x7f9f78101630 2024-03-28T02:21:59.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.849+0000 7f9f7cad2640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f9f640a29f0 0x7f9f640a4eb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9f780ff780 tx=0x7f9f68005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:21:59.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:21:59.852+0000 7f9f6e7fc640 1 -- 172.21.15.106:0/2550110560 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9f7409bb90 con 0x7f9f78101630 2024-03-28T02:21:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:59 smithi178 ceph-mon[36521]: pgmap v332: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:59 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/4015441394' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T02:21:59.872 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:21:59 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3673057762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:21:59.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:59 smithi153 ceph-mon[36228]: pgmap v332: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T02:21:59.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:59 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/4015441394' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T02:21:59.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:21:59 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3673057762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:22:00.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.030+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f78199230 con 0x7f9f78101630 2024-03-28T02:22:00.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.031+0000 7f9f6e7fc640 1 -- 172.21.15.106:0/2550110560 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+25414 (secure 0 0 0) 0x7f9f7409fc30 con 0x7f9f78101630 2024-03-28T02:22:00.025 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-28T02:22:00.025 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":102,"fsid":"d222a03c-eca6-11ee-95d0-87774f69a715","created":"2024-03-28T02:00:42.806452+0000","modified":"2024-03-28T02:21:13.433026+0000","last_up_change":"2024-03-28T02:07:55.703439+0000","last_in_change":"2024-03-28T02:07:34.658325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T02:04:06.605353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T02:08:46.952662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T02:08:47.976273+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T02:08:51.703592+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57987f85-e72f-4e07-b777-cc2243ffbbe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6803","nonce":1092015924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6805","nonce":1092015924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6809","nonce":1092015924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1092015924},{"type":"v1","addr":"172.21.15.106:6807","nonce":1092015924}]},"public_addr":"172.21.15.106:6803/1092015924","cluster_addr":"172.21.15.106:6805/1092015924","heartbeat_back_addr":"172.21.15.106:6809/1092015924","heartbeat_front_addr":"172.21.15.106:6807/1092015924","state":["exists","up"]},{"osd":1,"uuid":"45e0b5df-6da6-4f24-aaf4-2d1531425742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6801","nonce":2006532693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6803","nonce":2006532693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6807","nonce":2006532693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2006532693},{"type":"v1","addr":"172.21.15.153:6805","nonce":2006532693}]},"public_addr":"172.21.15.153:6801/2006532693","cluster_addr":"172.21.15.153:6803/2006532693","heartbeat_back_addr":"172.21.15.153:6807/2006532693","heartbeat_front_addr":"172.21.15.153:6805/2006532693","state":["exists","up"]},{"osd":2,"uuid":"44ba4df3-ae2f-4947-8eb9-e7d91e69bba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6801","nonce":3105645780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6803","nonce":3105645780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6807","nonce":3105645780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3105645780},{"type":"v1","addr":"172.21.15.178:6805","nonce":3105645780}]},"public_addr":"172.21.15.178:6801/3105645780","cluster_addr":"172.21.15.178:6803/3105645780","heartbeat_back_addr":"172.21.15.178:6807/3105645780","heartbeat_front_addr":"172.21.15.178:6805/3105645780","state":["exists","up"]},{"osd":3,"uuid":"21a1eac9-0d56-4b09-914d-7a70444b90ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6811","nonce":3263525761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6813","nonce":3263525761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6817","nonce":3263525761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":3263525761},{"type":"v1","addr":"172.21.15.106:6815","nonce":3263525761}]},"public_addr":"172.21.15.106:6811/3263525761","cluster_addr":"172.21.15.106:6813/3263525761","heartbeat_back_addr":"172.21.15.106:6817/3263525761","heartbeat_front_addr":"172.21.15.106:6815/3263525761","state":["exists","up"]},{"osd":4,"uuid":"ad7fd91b-fc97-464b-bd5b-56e9bcf663fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6809","nonce":3126868610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6811","nonce":3126868610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6815","nonce":3126868610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3126868610},{"type":"v1","addr":"172.21.15.153:6813","nonce":3126868610}]},"public_addr":"172.21.15.153:6809/3126868610","cluster_addr":"172.21.15.153:6811/3126868610","heartbeat_back_addr":"172.21.15.153:6815/3126868610","heartbeat_front_addr":"172.21.15.153:6813/3126868610","state":["exists","up"]},{"osd":5,"uuid":"8d9710de-3c2e-492e-8a73-e8483bc7b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6809","nonce":3071904021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6811","nonce":3071904021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6815","nonce":3071904021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3071904021},{"type":"v1","addr":"172.21.15.178:6813","nonce":3071904021}]},"public_addr":"172.21.15.178:6809/3071904021","cluster_addr":"172.21.15.178:6811/3071904021","heartbeat_back_addr":"172.21.15.178:6815/3071904021","heartbeat_front_addr":"172.21.15.178:6813/3071904021","state":["exists","up"]},{"osd":6,"uuid":"063f0b59-1636-4cff-8de5-176afcb329b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6819","nonce":233665499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6821","nonce":233665499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6825","nonce":233665499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":233665499},{"type":"v1","addr":"172.21.15.106:6823","nonce":233665499}]},"public_addr":"172.21.15.106:6819/233665499","cluster_addr":"172.21.15.106:6821/233665499","heartbeat_back_addr":"172.21.15.106:6825/233665499","heartbeat_front_addr":"172.21.15.106:6823/233665499","state":["exists","up"]},{"osd":7,"uuid":"77f29280-5275-445e-93a1-c93e64f17496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6817","nonce":1996966525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6819","nonce":1996966525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6823","nonce":1996966525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1996966525},{"type":"v1","addr":"172.21.15.153:6821","nonce":1996966525}]},"public_addr":"172.21.15.153:6817/1996966525","cluster_addr":"172.21.15.153:6819/1996966525","heartbeat_back_addr":"172.21.15.153:6823/1996966525","heartbeat_front_addr":"172.21.15.153:6821/1996966525","state":["exists","up"]},{"osd":8,"uuid":"46ffe893-a42e-4374-a83a-1316c70cd49d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6817","nonce":3897935183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6819","nonce":3897935183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6823","nonce":3897935183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3897935183},{"type":"v1","addr":"172.21.15.178:6821","nonce":3897935183}]},"public_addr":"172.21.15.178:6817/3897935183","cluster_addr":"172.21.15.178:6819/3897935183","heartbeat_back_addr":"172.21.15.178:6823/3897935183","heartbeat_front_addr":"172.21.15.178:6821/3897935183","state":["exists","up"]},{"osd":9,"uuid":"16014c57-a839-460f-b9e5-e1a336f53c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6827","nonce":4082459972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6829","nonce":4082459972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6833","nonce":4082459972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4082459972},{"type":"v1","addr":"172.21.15.106:6831","nonce":4082459972}]},"public_addr":"172.21.15.106:6827/4082459972","cluster_addr":"172.21.15.106:6829/4082459972","heartbeat_back_addr":"172.21.15.106:6833/4082459972","heartbeat_front_addr":"172.21.15.106:6831/4082459972","state":["exists","up"]},{"osd":10,"uuid":"5d65fed3-ba4f-4a3c-9469-c624012d5fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6825","nonce":1470813442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6827","nonce":1470813442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6831","nonce":1470813442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1470813442},{"type":"v1","addr":"172.21.15.153:6829","nonce":1470813442}]},"public_addr":"172.21.15.153:6825/1470813442","cluster_addr":"172.21.15.153:6827/1470813442","heartbeat_back_addr":"172.21.15.153:6831/1470813442","heartbeat_front_addr":"172.21.15.153:6829/1470813442","state":["exists","up"]},{"osd":11,"uuid":"37920df5-7610-4acf-98e8-33fad58f4d2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6825","nonce":3563501729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6827","nonce":3563501729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6831","nonce":3563501729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":3563501729},{"type":"v1","addr":"172.21.15.178:6829","nonce":3563501729}]},"public_addr":"172.21.15.178:6825/3563501729","cluster_addr":"172.21.15.178:6827/3563501729","heartbeat_back_addr":"172.21.15.178:6831/3563501729","heartbeat_front_addr":"172.21.15.178:6829/3563501729","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:16.279484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:03:39.685598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:03.295922+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:28.511873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:04:53.136840+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:18.422285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:05:43.453526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:08.881858+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:06:34.258162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:00.573706+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:27.105705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T02:07:53.277024+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6801/3501119323":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/2601893696":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/3655727768":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/11266040":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/2884389364":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/899620795":"2024-03-29T02:11:16.390724+0000","172.21.15.106:0/4050392782":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6800/3501119323":"2024-03-29T02:11:16.390724+0000","172.21.15.106:6800/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/103827077":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2664246824":"2024-03-29T02:11:16.390724+0000","172.21.15.106:6801/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:0/256234961":"2024-03-29T02:01:15.853032+0000","172.21.15.106:6800/305853075":"2024-03-29T02:01:42.267941+0000","172.21.15.106:6801/1510653090":"2024-03-29T02:01:15.853032+0000","172.21.15.106:0/2759478196":"2024-03-29T02:01:15.853032+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T02:22:00.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f9f640a29f0 msgr2=0x7f9f640a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f9f640a29f0 0x7f9f640a4eb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9f780ff780 tx=0x7f9f68005f90 comp rx=0 tx=0).stop 2024-03-28T02:22:00.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101630 msgr2=0x7f9f78197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101630 0x7f9f78197f30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9f7400e9f0 tx=0x7f9f7400eeb0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 shutdown_connections 2024-03-28T02:22:00.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f9f640a29f0 0x7f9f640a4eb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f78107640 0x7f9f7819c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f78101ff0 0x7f9f78198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 --2- 172.21.15.106:0/2550110560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f78101630 0x7f9f78197f30 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 >> 172.21.15.106:0/2550110560 conn(0x7f9f780fd140 msgr2=0x7f9f780fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:00.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 shutdown_connections 2024-03-28T02:22:00.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.033+0000 7f9f7f537640 1 -- 172.21.15.106:0/2550110560 wait complete. 2024-03-28T02:22:00.038 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.150+0000 7eff26762640 1 -- 172.21.15.106:0/3305255826 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 msgr2=0x7eff201069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.150+0000 7eff26762640 1 --2- 172.21.15.106:0/3305255826 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 0x7eff201069d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7eff10009900 tx=0x7eff1002bc30 comp rx=0 tx=0).stop 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.150+0000 7eff26762640 1 -- 172.21.15.106:0/3305255826 shutdown_connections 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.150+0000 7eff26762640 1 --2- 172.21.15.106:0/3305255826 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff20100ef0 0x7eff2010d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.150+0000 7eff26762640 1 --2- 172.21.15.106:0/3305255826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff20100530 0x7eff201009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.150+0000 7eff26762640 1 --2- 172.21.15.106:0/3305255826 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 0x7eff201069d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.150+0000 7eff26762640 1 -- 172.21.15.106:0/3305255826 >> 172.21.15.106:0/3305255826 conn(0x7eff200776a0 msgr2=0x7eff20077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.150+0000 7eff26762640 1 -- 172.21.15.106:0/3305255826 shutdown_connections 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.150+0000 7eff26762640 1 -- 172.21.15.106:0/3305255826 wait complete. 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff26762640 1 Processor -- start 2024-03-28T02:22:00.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff26762640 1 -- start start 2024-03-28T02:22:00.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff26762640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff20100530 0x7eff20197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:00.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff26762640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff20100ef0 0x7eff201983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff26762640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 0x7eff2019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff26762640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff201108b0 con 0x7eff201065f0 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff26762640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff20110730 con 0x7eff20100ef0 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff26762640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff20110a30 con 0x7eff20100530 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff24cff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 0x7eff2019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff1ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff20100530 0x7eff20197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff24cff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 0x7eff2019c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41606/0 (socket says 172.21.15.106:41606) 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff24cff640 1 -- 172.21.15.106:0/685733182 learned_addr learned my addr 172.21.15.106:0/685733182 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff1f7fe640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff20100ef0 0x7eff201983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff1ffff640 1 -- 172.21.15.106:0/685733182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff20100ef0 msgr2=0x7eff201983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.151+0000 7eff1ffff640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff20100ef0 0x7eff201983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff1ffff640 1 -- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 msgr2=0x7eff2019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff1ffff640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 0x7eff2019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff1ffff640 1 -- 172.21.15.106:0/685733182 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2019ce90 con 0x7eff20100530 2024-03-28T02:22:00.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff1f7fe640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff20100ef0 0x7eff201983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:22:00.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff24cff640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 0x7eff2019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:22:00.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff1ffff640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff20100530 0x7eff20197ea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7eff10000c00 tx=0x7eff100042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:00.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff1d7fa640 1 -- 172.21.15.106:0/685733182 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10041070 con 0x7eff20100530 2024-03-28T02:22:00.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2019d120 con 0x7eff20100530 2024-03-28T02:22:00.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff1d7fa640 1 -- 172.21.15.106:0/685733182 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff1002ccc0 con 0x7eff20100530 2024-03-28T02:22:00.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff1d7fa640 1 -- 172.21.15.106:0/685733182 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10003db0 con 0x7eff20100530 2024-03-28T02:22:00.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.152+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff201a4a10 con 0x7eff20100530 2024-03-28T02:22:00.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.153+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff20068490 con 0x7eff20100530 2024-03-28T02:22:00.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.154+0000 7eff1d7fa640 1 -- 172.21.15.106:0/685733182 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7eff10044070 con 0x7eff20100530 2024-03-28T02:22:00.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.155+0000 7eff1d7fa640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7eff080a2970 0x7eff080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:00.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.155+0000 7eff1f7fe640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7eff080a2970 0x7eff080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:00.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.155+0000 7eff1d7fa640 1 -- 172.21.15.106:0/685733182 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9504+0+0 (secure 0 0 0) 0x7eff100f0cf0 con 0x7eff20100530 2024-03-28T02:22:00.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.155+0000 7eff1f7fe640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7eff080a2970 0x7eff080a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7eff140095f0 tx=0x7eff14005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:00.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.159+0000 7eff1d7fa640 1 -- 172.21.15.106:0/685733182 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7eff10005670 con 0x7eff20100530 2024-03-28T02:22:00.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.380+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7eff201a4c10 con 0x7eff20100530 2024-03-28T02:22:00.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.387+0000 7eff1d7fa640 1 -- 172.21.15.106:0/685733182 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) v1 ==== 161+0+0 (secure 0 0 0) 0x7eff100bd100 con 0x7eff20100530 2024-03-28T02:22:00.381 INFO:teuthology.orchestra.run.smithi106.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T02:22:00.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7eff080a2970 msgr2=0x7eff080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7eff080a2970 0x7eff080a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7eff140095f0 tx=0x7eff14005f90 comp rx=0 tx=0).stop 2024-03-28T02:22:00.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff20100530 msgr2=0x7eff20197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff20100530 0x7eff20197ea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7eff10000c00 tx=0x7eff100042e0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 shutdown_connections 2024-03-28T02:22:00.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7eff080a2970 0x7eff080a4e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff201065f0 0x7eff2019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff20100ef0 0x7eff201983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 --2- 172.21.15.106:0/685733182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff20100530 0x7eff20197ea0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 >> 172.21.15.106:0/685733182 conn(0x7eff200776a0 msgr2=0x7eff200ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:00.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 shutdown_connections 2024-03-28T02:22:00.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.390+0000 7eff26762640 1 -- 172.21.15.106:0/685733182 wait complete. 2024-03-28T02:22:00.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T02:22:00.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 -- 172.21.15.106:0/271177246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa330069260 msgr2=0x7fa33010e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 --2- 172.21.15.106:0/271177246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa330069260 0x7fa33010e500 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa328009cd0 tx=0x7fa32802c0f0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 -- 172.21.15.106:0/271177246 shutdown_connections 2024-03-28T02:22:00.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 --2- 172.21.15.106:0/271177246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa330069260 0x7fa33010e500 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 --2- 172.21.15.106:0/271177246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3300688a0 0x7fa330068d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 --2- 172.21.15.106:0/271177246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3301076f0 0x7fa330107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 -- 172.21.15.106:0/271177246 >> 172.21.15.106:0/271177246 conn(0x7fa3300776a0 msgr2=0x7fa330077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:00.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 -- 172.21.15.106:0/271177246 shutdown_connections 2024-03-28T02:22:00.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 -- 172.21.15.106:0/271177246 wait complete. 2024-03-28T02:22:00.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.516+0000 7fa334bc6640 1 Processor -- start 2024-03-28T02:22:00.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa334bc6640 1 -- start start 2024-03-28T02:22:00.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa334bc6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3300688a0 0x7fa330197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:00.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa334bc6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa330069260 0x7fa330198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:00.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa334bc6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3301076f0 0x7fa33019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:00.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa334bc6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa330111a40 con 0x7fa3301076f0 2024-03-28T02:22:00.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa334bc6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3301118c0 con 0x7fa330069260 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa334bc6640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa330111bc0 con 0x7fa3300688a0 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32ed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3301076f0 0x7fa33019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32ed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3301076f0 0x7fa33019c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41640/0 (socket says 172.21.15.106:41640) 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32ed9d640 1 -- 172.21.15.106:0/3773810116 learned_addr learned my addr 172.21.15.106:0/3773810116 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32e59c640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3300688a0 0x7fa330197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32dd9b640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa330069260 0x7fa330198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32ed9d640 1 -- 172.21.15.106:0/3773810116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3300688a0 msgr2=0x7fa330197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32ed9d640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3300688a0 0x7fa330197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32ed9d640 1 -- 172.21.15.106:0/3773810116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa330069260 msgr2=0x7fa330198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32ed9d640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa330069260 0x7fa330198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32ed9d640 1 -- 172.21.15.106:0/3773810116 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33019ceb0 con 0x7fa3301076f0 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32dd9b640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa330069260 0x7fa330198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32e59c640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3300688a0 0x7fa330197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T02:22:00.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa32ed9d640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3301076f0 0x7fa33019c780 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa328009ca0 tx=0x7fa328032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:00.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa31f7fe640 1 -- 172.21.15.106:0/3773810116 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328042020 con 0x7fa3301076f0 2024-03-28T02:22:00.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.517+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33019d140 con 0x7fa3301076f0 2024-03-28T02:22:00.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.518+0000 7fa31f7fe640 1 -- 172.21.15.106:0/3773810116 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa328034640 con 0x7fa3301076f0 2024-03-28T02:22:00.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.518+0000 7fa31f7fe640 1 -- 172.21.15.106:0/3773810116 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328039040 con 0x7fa3301076f0 2024-03-28T02:22:00.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.518+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3301a4a30 con 0x7fa3301076f0 2024-03-28T02:22:00.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.518+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa330199210 con 0x7fa3301076f0 2024-03-28T02:22:00.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.520+0000 7fa31f7fe640 1 -- 172.21.15.106:0/3773810116 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fa328045020 con 0x7fa3301076f0 2024-03-28T02:22:00.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.520+0000 7fa31f7fe640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa3180a2970 0x7fa3180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:00.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.520+0000 7fa32e59c640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa3180a2970 0x7fa3180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:00.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.520+0000 7fa31f7fe640 1 -- 172.21.15.106:0/3773810116 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9846+0+0 (secure 0 0 0) 0x7fa3280f1790 con 0x7fa3301076f0 2024-03-28T02:22:00.514 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.521+0000 7fa32e59c640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa3180a2970 0x7fa3180a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa324009c20 tx=0x7fa324005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:00.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.528+0000 7fa31f7fe640 1 -- 172.21.15.106:0/3773810116 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa32804c020 con 0x7fa3301076f0 2024-03-28T02:22:00.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a[31961]: 2024-03-28T02:22:00.381+0000 7f0fbe269700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T02:22:00.583 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a[73998]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.583 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a[73998]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.583 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mds[74002]: -- 172.21.15.106:0/3892045742 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c0703884e0 con 0x55c07035f000 2024-03-28T02:22:00.583 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mds[74002]: -- 172.21.15.106:0/3892045742 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c07038ba40 con 0x55c07035f000 2024-03-28T02:22:00.583 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a[73998]: starting mds.a at 2024-03-28T02:22:00.584 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g[73652]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.584 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g[73652]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.584 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mds[73656]: -- 172.21.15.106:0/1404779079 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cfaaa85a40 con 0x55cfaaa59000 2024-03-28T02:22:00.584 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g[73652]: starting mds.g at 2024-03-28T02:22:00.733 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k[66226]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.733 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k[66226]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.733 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mds[66230]: -- 172.21.15.153:0/3259310083 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5607bd0dba40 con 0x5607bd0aec00 2024-03-28T02:22:00.733 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k[66226]: starting mds.k at 2024-03-28T02:22:00.734 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b[67179]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.734 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b[67179]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.734 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mds[67183]: -- 172.21.15.153:0/1126422358 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5644a7d45a40 con 0x5644a7d18c00 2024-03-28T02:22:00.734 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b[67179]: starting mds.b at 2024-03-28T02:22:00.734 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e[66549]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.734 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e[66549]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mds[66553]: -- 172.21.15.153:0/814182002 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5640f3465a40 con 0x5640f3438c00 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e[66549]: starting mds.e at 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h[66868]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h[66868]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mds[66872]: -- 172.21.15.153:0/3490822135 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eaf1c0ba40 con 0x55eaf1bde800 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h[66868]: starting mds.h at 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2947837685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/2550110560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/685733182' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T02:22:00.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mon[36228]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T02:22:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mon[36228]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T02:22:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mon[36228]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T02:22:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mon[36228]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T02:22:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mon[36228]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T02:22:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:00 smithi153 ceph-mon[36228]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T02:22:00.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:00.748+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa330102d70 con 0x7fa3301076f0 2024-03-28T02:22:00.872 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l[66730]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.872 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l[66730]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.872 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mds[66734]: -- 172.21.15.178:0/2750663966 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563f2e907a40 con 0x563f2e8db000 2024-03-28T02:22:00.872 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l[66730]: starting mds.l at 2024-03-28T02:22:00.873 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f[67037]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.873 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f[67037]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.873 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mds[67041]: -- 172.21.15.178:0/2591593655 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556d46a2a4e0 con 0x556d46a00c00 2024-03-28T02:22:00.873 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mds[67041]: -- 172.21.15.178:0/2591593655 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556d46a2da40 con 0x556d46a00c00 2024-03-28T02:22:00.873 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f[67037]: starting mds.f at 2024-03-28T02:22:00.873 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i[67656]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.873 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i[67656]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.873 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mds[67660]: -- 172.21.15.178:0/1455567911 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559932ef9a40 con 0x559932eccc00 2024-03-28T02:22:00.873 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i[67656]: starting mds.i at 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c[67347]: ignoring --setuser ceph since I am not root 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c[67347]: ignoring --setgroup ceph since I am not root 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mds[67351]: -- 172.21.15.178:0/3642759703 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5589a0517a40 con 0x5589a04ea800 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c[67347]: starting mds.c at 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2947837685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/2550110560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/685733182' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mon[36521]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mon[36521]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mon[36521]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T02:22:00.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mon[36521]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T02:22:00.875 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mon[36521]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T02:22:00.875 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:00 smithi178 ceph-mon[36521]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T02:22:01.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2947837685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T02:22:01.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/2550110560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T02:22:01.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/685733182' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T02:22:01.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mon[31965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T02:22:01.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mon[31965]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T02:22:01.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mon[31965]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T02:22:01.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mon[31965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T02:22:01.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mon[31965]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T02:22:01.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:00 smithi106 ceph-mon[31965]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T02:22:01.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.388+0000 7fa31f7fe640 1 -- 172.21.15.106:0/3773810116 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa3280be150 con 0x7fa3301076f0 2024-03-28T02:22:01.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa3180a2970 msgr2=0x7fa3180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:01.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa3180a2970 0x7fa3180a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa324009c20 tx=0x7fa324005f90 comp rx=0 tx=0).stop 2024-03-28T02:22:01.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3301076f0 msgr2=0x7fa33019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:01.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3301076f0 0x7fa33019c780 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa328009ca0 tx=0x7fa328032ec0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 shutdown_connections 2024-03-28T02:22:01.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7fa3180a2970 0x7fa3180a4e30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3301076f0 0x7fa33019c780 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa330069260 0x7fa330198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 --2- 172.21.15.106:0/3773810116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3300688a0 0x7fa330197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 >> 172.21.15.106:0/3773810116 conn(0x7fa3300776a0 msgr2=0x7fa3300fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:01.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 shutdown_connections 2024-03-28T02:22:01.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.391+0000 7fa334bc6640 1 -- 172.21.15.106:0/3773810116 wait complete. 2024-03-28T02:22:01.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T02:22:01.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 -- 172.21.15.106:0/151210250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 msgr2=0x7f8a241009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:01.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/151210250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 0x7f8a241009c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009980 tx=0x7f8a2002bc50 comp rx=0 tx=0).stop 2024-03-28T02:22:01.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 -- 172.21.15.106:0/151210250 shutdown_connections 2024-03-28T02:22:01.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/151210250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a24100f00 0x7f8a2410d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/151210250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 0x7f8a241009c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/151210250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a24106600 0x7f8a241069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 -- 172.21.15.106:0/151210250 >> 172.21.15.106:0/151210250 conn(0x7f8a24077750 msgr2=0x7f8a24077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 -- 172.21.15.106:0/151210250 shutdown_connections 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 -- 172.21.15.106:0/151210250 wait complete. 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 Processor -- start 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 -- start start 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.511+0000 7f8a2be4b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 0x7f8a2419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a2be4b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a24100f00 0x7f8a2419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a2be4b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a24106600 0x7f8a24195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a2be4b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2410f840 con 0x7f8a24100540 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a2be4b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2410f6c0 con 0x7f8a24106600 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a2be4b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2410f9c0 con 0x7f8a24100f00 2024-03-28T02:22:01.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a29be7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 0x7f8a2419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a29be7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 0x7f8a2419b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41656/0 (socket says 172.21.15.106:41656) 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a29be7640 1 -- 172.21.15.106:0/1820147739 learned_addr learned my addr 172.21.15.106:0/1820147739 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a293e6640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a24100f00 0x7f8a2419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a2a3e8640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a24106600 0x7f8a24195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a29be7640 1 -- 172.21.15.106:0/1820147739 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a24100f00 msgr2=0x7f8a2419bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a29be7640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a24100f00 0x7f8a2419bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a29be7640 1 -- 172.21.15.106:0/1820147739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a24106600 msgr2=0x7f8a24195a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a29be7640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a24106600 0x7f8a24195a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a29be7640 1 -- 172.21.15.106:0/1820147739 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24196180 con 0x7f8a24100540 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a2a3e8640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a24106600 0x7f8a24195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a29be7640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 0x7f8a2419b850 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400d950 tx=0x7f8a1400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a1affd640 1 -- 172.21.15.106:0/1820147739 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14017070 con 0x7f8a24100540 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a24196470 con 0x7f8a24100540 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a1affd640 1 -- 172.21.15.106:0/1820147739 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a14009c30 con 0x7f8a24100540 2024-03-28T02:22:01.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a1affd640 1 -- 172.21.15.106:0/1820147739 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1400f6b0 con 0x7f8a24100540 2024-03-28T02:22:01.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.512+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a241a2710 con 0x7f8a24100540 2024-03-28T02:22:01.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.513+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a24068490 con 0x7f8a24100540 2024-03-28T02:22:01.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.515+0000 7f8a1affd640 1 -- 172.21.15.106:0/1820147739 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f8a140109d0 con 0x7f8a24100540 2024-03-28T02:22:01.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.515+0000 7f8a1affd640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f8a100a2970 0x7f8a100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:01.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.515+0000 7f8a293e6640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f8a100a2970 0x7f8a100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:01.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.516+0000 7f8a1affd640 1 -- 172.21.15.106:0/1820147739 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9846+0+0 (secure 0 0 0) 0x7f8a140ceff0 con 0x7f8a24100540 2024-03-28T02:22:01.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.516+0000 7f8a293e6640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f8a100a2970 0x7f8a100a4e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8a20006010 tx=0x7f8a20005b50 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:01.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.519+0000 7f8a1affd640 1 -- 172.21.15.106:0/1820147739 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8a14009de0 con 0x7f8a24100540 2024-03-28T02:22:01.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:01.695+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8a24108970 con 0x7f8a24100540 2024-03-28T02:22:01.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: pgmap v334: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T02:22:01.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3773810116' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: Cluster is now healthy 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3773810116' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6835/1266757317,v1:172.21.15.178:6839/1266757317] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6834/4053050246,v1:172.21.15.178:6836/4053050246] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.106:6838/192041722,v1:172.21.15.106:6840/192041722] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.106:6839/3992807492,v1:172.21.15.106:6841/3992807492] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6836/364410534,v1:172.21.15.153:6838/364410534] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6832/404211926,v1:172.21.15.153:6833/404211926] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6837/410173384,v1:172.21.15.153:6839/410173384] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.153:6834/1857795748,v1:172.21.15.153:6835/1857795748] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6832/1091663278,v1:172.21.15.178:6838/1091663278] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: mds.? [v2:172.21.15.178:6833/2375985099,v1:172.21.15.178:6837/2375985099] up:boot 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: fsmap 12 up:standby 2024-03-28T02:22:01.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T02:22:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T02:22:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T02:22:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T02:22:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T02:22:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T02:22:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T02:22:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T02:22:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T02:22:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:01 smithi153 ceph-mon[36228]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T02:22:02.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: pgmap v334: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T02:22:02.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3773810116' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T02:22:02.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T02:22:02.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T02:22:02.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: Cluster is now healthy 2024-03-28T02:22:02.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3773810116' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6835/1266757317,v1:172.21.15.178:6839/1266757317] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6834/4053050246,v1:172.21.15.178:6836/4053050246] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.106:6838/192041722,v1:172.21.15.106:6840/192041722] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.106:6839/3992807492,v1:172.21.15.106:6841/3992807492] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6836/364410534,v1:172.21.15.153:6838/364410534] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6832/404211926,v1:172.21.15.153:6833/404211926] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6837/410173384,v1:172.21.15.153:6839/410173384] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.153:6834/1857795748,v1:172.21.15.153:6835/1857795748] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6832/1091663278,v1:172.21.15.178:6838/1091663278] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: mds.? [v2:172.21.15.178:6833/2375985099,v1:172.21.15.178:6837/2375985099] up:boot 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: fsmap 12 up:standby 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T02:22:02.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T02:22:02.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T02:22:02.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T02:22:02.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T02:22:02.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:01 smithi106 ceph-mon[31965]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T02:22:02.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: pgmap v334: 129 pgs: 129 active+clean; 4.4 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T02:22:02.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3773810116' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T02:22:02.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T02:22:02.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T02:22:02.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: Cluster is now healthy 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3773810116' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6835/1266757317,v1:172.21.15.178:6839/1266757317] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6834/4053050246,v1:172.21.15.178:6836/4053050246] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.106:6838/192041722,v1:172.21.15.106:6840/192041722] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.106:6839/3992807492,v1:172.21.15.106:6841/3992807492] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6836/364410534,v1:172.21.15.153:6838/364410534] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6832/404211926,v1:172.21.15.153:6833/404211926] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6837/410173384,v1:172.21.15.153:6839/410173384] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.153:6834/1857795748,v1:172.21.15.153:6835/1857795748] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6832/1091663278,v1:172.21.15.178:6838/1091663278] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: mds.? [v2:172.21.15.178:6833/2375985099,v1:172.21.15.178:6837/2375985099] up:boot 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: fsmap 12 up:standby 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T02:22:02.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T02:22:02.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T02:22:02.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T02:22:02.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T02:22:02.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T02:22:02.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T02:22:02.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:01 smithi178 ceph-mon[36521]: from='mgr.25315 172.21.15.106:0/416488963' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T02:22:02.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.638+0000 7f8a1affd640 1 -- 172.21.15.106:0/1820147739 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8a1409b800 con 0x7f8a24100540 2024-03-28T02:22:02.632 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'cephfs_metadata' removed 2024-03-28T02:22:02.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.640+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f8a100a2970 msgr2=0x7f8a100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:02.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.640+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f8a100a2970 0x7f8a100a4e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8a20006010 tx=0x7f8a20005b50 comp rx=0 tx=0).stop 2024-03-28T02:22:02.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.640+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 msgr2=0x7f8a2419b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:02.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.640+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 0x7f8a2419b850 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400d950 tx=0x7f8a1400de10 comp rx=0 tx=0).stop 2024-03-28T02:22:02.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.641+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 shutdown_connections 2024-03-28T02:22:02.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.641+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f8a100a2970 0x7f8a100a4e30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:02.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.641+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a24106600 0x7f8a24195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:02.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.641+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a24100f00 0x7f8a2419bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:02.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.641+0000 7f8a2be4b640 1 --2- 172.21.15.106:0/1820147739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a24100540 0x7f8a2419b850 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:02.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.641+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 >> 172.21.15.106:0/1820147739 conn(0x7f8a24077750 msgr2=0x7f8a240ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:02.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.641+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 shutdown_connections 2024-03-28T02:22:02.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.641+0000 7f8a2be4b640 1 -- 172.21.15.106:0/1820147739 wait complete. 2024-03-28T02:22:02.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T02:22:02.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.767+0000 7f82442dc640 1 -- 172.21.15.106:0/2049584085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c1065f0 msgr2=0x7f823c1069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:02.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.767+0000 7f82442dc640 1 --2- 172.21.15.106:0/2049584085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c1065f0 0x7f823c1069d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8238009900 tx=0x7f823802bc30 comp rx=0 tx=0).stop 2024-03-28T02:22:02.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 -- 172.21.15.106:0/2049584085 shutdown_connections 2024-03-28T02:22:02.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 --2- 172.21.15.106:0/2049584085 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c100ef0 0x7f823c10d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:02.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 --2- 172.21.15.106:0/2049584085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c100530 0x7f823c1009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:02.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 --2- 172.21.15.106:0/2049584085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c1065f0 0x7f823c1069d0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:02.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 -- 172.21.15.106:0/2049584085 >> 172.21.15.106:0/2049584085 conn(0x7f823c0776a0 msgr2=0x7f823c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:02.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 -- 172.21.15.106:0/2049584085 shutdown_connections 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 -- 172.21.15.106:0/2049584085 wait complete. 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 Processor -- start 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 -- start start 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c100530 0x7f823c10de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c100ef0 0x7f823c10e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c1065f0 0x7f823c10e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c111a50 con 0x7f823c100530 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c1118d0 con 0x7f823c100ef0 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.768+0000 7f82442dc640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c111bd0 con 0x7f823c1065f0 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242078640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c100530 0x7f823c10de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242078640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c100530 0x7f823c10de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41686/0 (socket says 172.21.15.106:41686) 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242078640 1 -- 172.21.15.106:0/3903889188 learned_addr learned my addr 172.21.15.106:0/3903889188 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8241877640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c100ef0 0x7f823c10e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242879640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c1065f0 0x7f823c10e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:02.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242078640 1 -- 172.21.15.106:0/3903889188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c1065f0 msgr2=0x7f823c10e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242078640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c1065f0 0x7f823c10e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242078640 1 -- 172.21.15.106:0/3903889188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c100ef0 msgr2=0x7f823c10e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242078640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c100ef0 0x7f823c10e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242078640 1 -- 172.21.15.106:0/3903889188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c1a6470 con 0x7f823c100530 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8241877640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c100ef0 0x7f823c10e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f8242078640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c100530 0x7f823c10de40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8238005940 tx=0x7f82380042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f82337fe640 1 -- 172.21.15.106:0/3903889188 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238041070 con 0x7f823c100530 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f82337fe640 1 -- 172.21.15.106:0/3903889188 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f823802ccc0 con 0x7f823c100530 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823c1a6700 con 0x7f823c100530 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f82337fe640 1 -- 172.21.15.106:0/3903889188 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238003d80 con 0x7f823c100530 2024-03-28T02:22:02.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.769+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823c1a6bf0 con 0x7f823c100530 2024-03-28T02:22:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.770+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f823c068490 con 0x7f823c100530 2024-03-28T02:22:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.771+0000 7f82337fe640 1 -- 172.21.15.106:0/3903889188 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f8238044070 con 0x7f823c100530 2024-03-28T02:22:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.772+0000 7f82337fe640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f82280a2970 0x7f82280a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.772+0000 7f8241877640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f82280a2970 0x7f82280a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:02.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.772+0000 7f82337fe640 1 -- 172.21.15.106:0/3903889188 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9321+0+0 (secure 0 0 0) 0x7f82380f0e70 con 0x7f823c100530 2024-03-28T02:22:02.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.772+0000 7f8241877640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f82280a2970 0x7f82280a4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f823c10f8d0 tx=0x7f822c005f90 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:02.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.775+0000 7f82337fe640 1 -- 172.21.15.106:0/3903889188 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f82380bd9c0 con 0x7f823c100530 2024-03-28T02:22:02.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:02 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1820147739' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T02:22:02.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:02.950+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f823c1089e0 con 0x7f823c100530 2024-03-28T02:22:03.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:02 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1820147739' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T02:22:03.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:02 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1820147739' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T02:22:03.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.639+0000 7f82337fe640 1 -- 172.21.15.106:0/3903889188 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) v1 ==== 169+0+0 (secure 0 0 0) 0x7f82380bd2c0 con 0x7f823c100530 2024-03-28T02:22:03.633 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'cephfs_data' removed 2024-03-28T02:22:03.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f82280a2970 msgr2=0x7f82280a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:03.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f82280a2970 0x7f82280a4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f823c10f8d0 tx=0x7f822c005f90 comp rx=0 tx=0).stop 2024-03-28T02:22:03.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c100530 msgr2=0x7f823c10de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:03.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c100530 0x7f823c10de40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8238005940 tx=0x7f82380042f0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 shutdown_connections 2024-03-28T02:22:03.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7f82280a2970 0x7f82280a4e30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c1065f0 0x7f823c10e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c100ef0 0x7f823c10e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 --2- 172.21.15.106:0/3903889188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f823c100530 0x7f823c10de40 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 >> 172.21.15.106:0/3903889188 conn(0x7f823c0776a0 msgr2=0x7f823c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:03.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 shutdown_connections 2024-03-28T02:22:03.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.642+0000 7f82442dc640 1 -- 172.21.15.106:0/3903889188 wait complete. 2024-03-28T02:22:03.651 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-28T02:22:03.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.770+0000 7ff532916640 1 -- 172.21.15.106:0/1939954475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 msgr2=0x7ff52c107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:03.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.770+0000 7ff532916640 1 --2- 172.21.15.106:0/1939954475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 0x7ff52c107ad0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009930 tx=0x7ff51c02bc30 comp rx=0 tx=0).stop 2024-03-28T02:22:03.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 -- 172.21.15.106:0/1939954475 shutdown_connections 2024-03-28T02:22:03.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 --2- 172.21.15.106:0/1939954475 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff52c06a400 0x7ff52c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 --2- 172.21.15.106:0/1939954475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c069a40 0x7ff52c069ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 --2- 172.21.15.106:0/1939954475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 0x7ff52c107ad0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 -- 172.21.15.106:0/1939954475 >> 172.21.15.106:0/1939954475 conn(0x7ff52c0776d0 msgr2=0x7ff52c077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:03.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 -- 172.21.15.106:0/1939954475 shutdown_connections 2024-03-28T02:22:03.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 -- 172.21.15.106:0/1939954475 wait complete. 2024-03-28T02:22:03.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 Processor -- start 2024-03-28T02:22:03.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 -- start start 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff52c069a40 0x7ff52c197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c06a400 0x7ff52c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 0x7ff52c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c111a40 con 0x7ff52c1076f0 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1118c0 con 0x7ff52c06a400 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.771+0000 7ff532916640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c111bc0 con 0x7ff52c069a40 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff530eb3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 0x7ff52c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff530eb3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 0x7ff52c19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41690/0 (socket says 172.21.15.106:41690) 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff530eb3640 1 -- 172.21.15.106:0/3944571573 learned_addr learned my addr 172.21.15.106:0/3944571573 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff52b7fe640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c06a400 0x7ff52c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff52bfff640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff52c069a40 0x7ff52c197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:03.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff530eb3640 1 -- 172.21.15.106:0/3944571573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff52c069a40 msgr2=0x7ff52c197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff530eb3640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff52c069a40 0x7ff52c197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff530eb3640 1 -- 172.21.15.106:0/3944571573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c06a400 msgr2=0x7ff52c198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff530eb3640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c06a400 0x7ff52c198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff530eb3640 1 -- 172.21.15.106:0/3944571573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c19cdb0 con 0x7ff52c1076f0 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff530eb3640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 0x7ff52c19c680 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff52400eaa0 tx=0x7ff52400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff5297fa640 1 -- 172.21.15.106:0/3944571573 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff524004d20 con 0x7ff52c1076f0 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52c19d0a0 con 0x7ff52c1076f0 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff5297fa640 1 -- 172.21.15.106:0/3944571573 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff524005020 con 0x7ff52c1076f0 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff5297fa640 1 -- 172.21.15.106:0/3944571573 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5240116d0 con 0x7ff52c1076f0 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.772+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52c1a4930 con 0x7ff52c1076f0 2024-03-28T02:22:03.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.773+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52c067c00 con 0x7ff52c1076f0 2024-03-28T02:22:03.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.774+0000 7ff5297fa640 1 -- 172.21.15.106:0/3944571573 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 25) v1 ==== 137422+0+0 (secure 0 0 0) 0x7ff5240051d0 con 0x7ff52c1076f0 2024-03-28T02:22:03.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.775+0000 7ff5297fa640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff5140a2970 0x7ff5140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T02:22:03.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.775+0000 7ff52bfff640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff5140a2970 0x7ff5140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T02:22:03.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.775+0000 7ff5297fa640 1 -- 172.21.15.106:0/3944571573 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8754+0+0 (secure 0 0 0) 0x7ff5240cf4f0 con 0x7ff52c1076f0 2024-03-28T02:22:03.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.775+0000 7ff52bfff640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff5140a2970 0x7ff5140a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff51c006010 tx=0x7ff51c02c730 comp rx=0 tx=0).ready entity=mgr.25315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T02:22:03.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.779+0000 7ff5297fa640 1 -- 172.21.15.106:0/3944571573 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff524016030 con 0x7ff52c1076f0 2024-03-28T02:22:03.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:03.952+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff52c1a4d30 con 0x7ff52c1076f0 2024-03-28T02:22:03.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:03 smithi153 ceph-mon[36228]: pgmap v335: 129 pgs: 129 active+clean; 4.4 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T02:22:03.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:03 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/1820147739' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T02:22:03.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:03 smithi153 ceph-mon[36228]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T02:22:03.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:03 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3903889188' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T02:22:04.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:03 smithi106 ceph-mon[31965]: pgmap v335: 129 pgs: 129 active+clean; 4.4 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T02:22:04.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:03 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/1820147739' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T02:22:04.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:03 smithi106 ceph-mon[31965]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T02:22:04.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:03 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3903889188' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T02:22:04.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:03 smithi178 ceph-mon[36521]: pgmap v335: 129 pgs: 129 active+clean; 4.4 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T02:22:04.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:03 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/1820147739' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T02:22:04.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:03 smithi178 ceph-mon[36521]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T02:22:04.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:03 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3903889188' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T02:22:04.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.653+0000 7ff5297fa640 1 -- 172.21.15.106:0/3944571573 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v106) v1 ==== 178+0+0 (secure 0 0 0) 0x7ff52409c160 con 0x7ff52c1076f0 2024-03-28T02:22:04.647 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'cephfs_data_ec' removed 2024-03-28T02:22:04.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff5140a2970 msgr2=0x7ff5140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:04.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff5140a2970 0x7ff5140a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff51c006010 tx=0x7ff51c02c730 comp rx=0 tx=0).stop 2024-03-28T02:22:04.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 msgr2=0x7ff52c19c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T02:22:04.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 0x7ff52c19c680 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff52400eaa0 tx=0x7ff52400eea0 comp rx=0 tx=0).stop 2024-03-28T02:22:04.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 shutdown_connections 2024-03-28T02:22:04.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:6800/2358705651,v1:172.21.15.106:6801/2358705651] conn(0x7ff5140a2970 0x7ff5140a4e30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:04.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff52c1076f0 0x7ff52c19c680 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:04.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c06a400 0x7ff52c198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:04.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 --2- 172.21.15.106:0/3944571573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff52c069a40 0x7ff52c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T02:22:04.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 >> 172.21.15.106:0/3944571573 conn(0x7ff52c0776d0 msgr2=0x7ff52c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T02:22:04.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 shutdown_connections 2024-03-28T02:22:04.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-28T02:22:04.657+0000 7ff532916640 1 -- 172.21.15.106:0/3944571573 wait complete. 2024-03-28T02:22:04.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T02:22:04.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T02:22:04.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T02:22:04.769 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-28T02:22:04.769 DEBUG:teuthology.orchestra.run.smithi106:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T02:22:04.785 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T02:22:04.802 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T02:22:04.826 INFO:tasks.cephadm:Stopping all daemons... 2024-03-28T02:22:04.826 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-28T02:22:04.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.a 2024-03-28T02:22:04.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:04 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3903889188' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T02:22:04.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:04 smithi153 ceph-mon[36228]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T02:22:04.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:04 smithi153 ceph-mon[36228]: from='client.? 172.21.15.106:0/3944571573' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T02:22:04.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:04 smithi153 ceph-mon[36228]: pgmap v338: 33 pgs: 33 active+clean; 577 KiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:22:05.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:04 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3903889188' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T02:22:05.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:04 smithi106 ceph-mon[31965]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T02:22:05.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:04 smithi106 ceph-mon[31965]: from='client.? 172.21.15.106:0/3944571573' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T02:22:05.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:04 smithi106 ceph-mon[31965]: pgmap v338: 33 pgs: 33 active+clean; 577 KiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:22:05.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:04 smithi106 systemd[1]: Stopping Ceph mon.a for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:05.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:04 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3903889188' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T02:22:05.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:04 smithi178 ceph-mon[36521]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T02:22:05.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:04 smithi178 ceph-mon[36521]: from='client.? 172.21.15.106:0/3944571573' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T02:22:05.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:04 smithi178 ceph-mon[36521]: pgmap v338: 33 pgs: 33 active+clean; 577 KiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T02:22:05.380 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:05 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a[31961]: 2024-03-28T02:22:05.122+0000 7f0fc3a74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:05.381 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:05 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a[31961]: 2024-03-28T02:22:05.122+0000 7f0fc3a74700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-28T02:22:05.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:05 smithi106 podman[92959]: 2024-03-28 02:22:05.388356836 +0000 UTC m=+0.364453309 container died f36f9e681e52ccbc0fdafc94c356efa07f9b8be9d86ca8a88561b3a8b55da130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:22:06.276 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:05 smithi106 podman[92959]: 2024-03-28 02:22:05.957190356 +0000 UTC m=+0.933286841 container cleanup f36f9e681e52ccbc0fdafc94c356efa07f9b8be9d86ca8a88561b3a8b55da130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T02:22:06.276 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 28 02:22:05 smithi106 bash[92959]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-a 2024-03-28T02:22:06.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.a.service' 2024-03-28T02:22:06.467 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:06.467 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-28T02:22:06.467 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-28T02:22:06.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.b 2024-03-28T02:22:06.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:06 smithi153 systemd[1]: Stopping Ceph mon.b for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:07.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:06 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-b[36224]: 2024-03-28T02:22:06.794+0000 7f0617559700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:07.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:06 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-b[36224]: 2024-03-28T02:22:06.794+0000 7f0617559700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-28T02:22:07.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:06 smithi153 podman[72618]: 2024-03-28 02:22:06.917552772 +0000 UTC m=+0.247118379 container died 03b91e3f4431c09af177c044a7f03176d5fe1da3ddafb28d519bf24ec422f2e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T02:22:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:07 smithi153 podman[72618]: 2024-03-28 02:22:07.311881643 +0000 UTC m=+0.641447269 container cleanup 03b91e3f4431c09af177c044a7f03176d5fe1da3ddafb28d519bf24ec422f2e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T02:22:07.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 28 02:22:07 smithi153 bash[72618]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-b 2024-03-28T02:22:07.727 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.b.service' 2024-03-28T02:22:07.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:07.796 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-28T02:22:07.796 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-28T02:22:07.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.c 2024-03-28T02:22:08.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:07 smithi178 systemd[1]: Stopping Ceph mon.c for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:08.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:08 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-c[36517]: 2024-03-28T02:22:08.122+0000 7fa05ce24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:08.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:08 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-c[36517]: 2024-03-28T02:22:08.122+0000 7fa05ce24700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-28T02:22:08.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:08 smithi178 podman[72999]: 2024-03-28 02:22:08.214685388 +0000 UTC m=+0.210230281 container died 91173284394dd742aab17b84bc85eea5ec4db8b8160425720369313671e7a892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T02:22:08.862 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:08 smithi178 podman[72999]: 2024-03-28 02:22:08.608274501 +0000 UTC m=+0.603819391 container cleanup 91173284394dd742aab17b84bc85eea5ec4db8b8160425720369313671e7a892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T02:22:08.862 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 28 02:22:08 smithi178 bash[72999]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mon-c 2024-03-28T02:22:09.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mon.c.service' 2024-03-28T02:22:09.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:09.084 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-28T02:22:09.084 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-28T02:22:09.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.x 2024-03-28T02:22:09.333 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:22:09 smithi106 systemd[1]: Stopping Ceph mgr.x for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:09.681 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:22:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:22:09.419+0000 7f11d64db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:09.681 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:22:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x[32194]: 2024-03-28T02:22:09.419+0000 7f11d64db700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-28T02:22:09.682 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:22:09 smithi106 podman[93080]: 2024-03-28 02:22:09.451604426 +0000 UTC m=+0.162317406 container died 392dee3cdfe634b560e74602ab365baebe3a4cd95d937017103599107a3f1cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:22:09.991 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:22:09 smithi106 podman[93080]: 2024-03-28 02:22:09.796887538 +0000 UTC m=+0.507600571 container cleanup 392dee3cdfe634b560e74602ab365baebe3a4cd95d937017103599107a3f1cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T02:22:09.991 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Mar 28 02:22:09 smithi106 bash[93080]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-x 2024-03-28T02:22:10.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-28T02:22:10.245 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:10.245 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-28T02:22:10.245 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-28T02:22:10.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.y 2024-03-28T02:22:10.580 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:22:10 smithi153 systemd[1]: Stopping Ceph mgr.y for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:10.869 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:22:10 smithi153 podman[72741]: 2024-03-28 02:22:10.587372136 +0000 UTC m=+0.144660420 container died 9442b8a6cda58dd51d258aaaaecc0611502e41a732cadb555b1b6052889f0dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T02:22:11.220 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:22:10 smithi153 podman[72741]: 2024-03-28 02:22:10.984232255 +0000 UTC m=+0.541520525 container cleanup 9442b8a6cda58dd51d258aaaaecc0611502e41a732cadb555b1b6052889f0dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T02:22:11.220 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 28 02:22:10 smithi153 bash[72741]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-y 2024-03-28T02:22:11.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-28T02:22:11.395 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:11.395 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-28T02:22:11.396 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-28T02:22:11.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.z 2024-03-28T02:22:11.727 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:22:11 smithi178 systemd[1]: Stopping Ceph mgr.z for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:11.999 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:22:11 smithi178 podman[73119]: 2024-03-28 02:22:11.73406218 +0000 UTC m=+0.142524117 container died 334ab3dddb836c71c2fb9ee4bdf49338a3c7bac7e8fdcbb613de51cd988e9490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T02:22:12.372 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:22:12 smithi178 podman[73119]: 2024-03-28 02:22:12.114118661 +0000 UTC m=+0.522580584 container cleanup 334ab3dddb836c71c2fb9ee4bdf49338a3c7bac7e8fdcbb613de51cd988e9490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:22:12.372 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 28 02:22:12 smithi178 bash[73119]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mgr-z 2024-03-28T02:22:12.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-28T02:22:12.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:12.566 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-28T02:22:12.567 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-28T02:22:12.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.0 2024-03-28T02:22:12.895 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:12 smithi106 systemd[1]: Stopping Ceph osd.0 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:13.333 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0[46711]: 2024-03-28T02:22:12.900+0000 7fccf7252700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:13.333 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0[46711]: 2024-03-28T02:22:12.900+0000 7fccf7252700 -1 osd.0 106 *** Got signal Terminated *** 2024-03-28T02:22:13.333 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0[46711]: 2024-03-28T02:22:12.900+0000 7fccf7252700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:22:18.538 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:18 smithi106 podman[93202]: 2024-03-28 02:22:18.269756336 +0000 UTC m=+5.499618028 container died 5b85f5a59426149b43d44c3c1d4abae053b372891b72a126110a463560007396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-28T02:22:18.830 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:18 smithi106 podman[93202]: 2024-03-28 02:22:18.601912207 +0000 UTC m=+5.831773904 container cleanup 5b85f5a59426149b43d44c3c1d4abae053b372891b72a126110a463560007396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:22:18.830 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:18 smithi106 bash[93202]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0 2024-03-28T02:22:19.083 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:18 smithi106 podman[93222]: 2024-03-28 02:22:18.837072628 +0000 UTC m=+0.570092703 container remove 5b85f5a59426149b43d44c3c1d4abae053b372891b72a126110a463560007396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T02:22:19.525 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:19 smithi106 podman[93312]: 2024-03-28 02:22:19.15568638 +0000 UTC m=+0.017725010 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:22:19.525 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:19 smithi106 podman[93312]: 2024-03-28 02:22:19.263098405 +0000 UTC m=+0.125137046 container create 75b9293597d13e94831d9960aa138551fedefc0577986f6a770ad4f7b1cebe33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True) 2024-03-28T02:22:19.525 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:19 smithi106 podman[93312]: 2024-03-28 02:22:19.532473286 +0000 UTC m=+0.394511916 container init 75b9293597d13e94831d9960aa138551fedefc0577986f6a770ad4f7b1cebe33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T02:22:19.825 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:19 smithi106 podman[93312]: 2024-03-28 02:22:19.535833469 +0000 UTC m=+0.397872099 container start 75b9293597d13e94831d9960aa138551fedefc0577986f6a770ad4f7b1cebe33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T02:22:19.826 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:19 smithi106 podman[93312]: 2024-03-28 02:22:19.590908698 +0000 UTC m=+0.452947331 container attach 75b9293597d13e94831d9960aa138551fedefc0577986f6a770ad4f7b1cebe33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T02:22:20.083 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 28 02:22:19 smithi106 podman[93312]: 2024-03-28 02:22:19.833108582 +0000 UTC m=+0.695147218 container died 75b9293597d13e94831d9960aa138551fedefc0577986f6a770ad4f7b1cebe33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:22:20.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.0.service' 2024-03-28T02:22:20.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:20.515 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-28T02:22:20.515 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-28T02:22:20.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.1 2024-03-28T02:22:20.823 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:20 smithi153 systemd[1]: Stopping Ceph osd.1 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:21.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:20 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1[40804]: 2024-03-28T02:22:20.829+0000 7f40ad010700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:21.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:20 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1[40804]: 2024-03-28T02:22:20.829+0000 7f40ad010700 -1 osd.1 106 *** Got signal Terminated *** 2024-03-28T02:22:21.234 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:20 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1[40804]: 2024-03-28T02:22:20.829+0000 7f40ad010700 -1 osd.1 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:22:26.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:26 smithi153 podman[72863]: 2024-03-28 02:22:26.364779355 +0000 UTC m=+5.647709103 container died bbb28c88602539f9763cb72affd670e8f82dc857abc4b11c648d82bf8cfc9eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T02:22:26.983 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:26 smithi153 podman[72863]: 2024-03-28 02:22:26.788445515 +0000 UTC m=+6.071375262 container cleanup bbb28c88602539f9763cb72affd670e8f82dc857abc4b11c648d82bf8cfc9eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T02:22:26.984 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:26 smithi153 bash[72863]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1 2024-03-28T02:22:27.442 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:27 smithi153 podman[72880]: 2024-03-28 02:22:27.023525106 +0000 UTC m=+0.658955762 container remove bbb28c88602539f9763cb72affd670e8f82dc857abc4b11c648d82bf8cfc9eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T02:22:27.728 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:27 smithi153 podman[72972]: 2024-03-28 02:22:27.351839054 +0000 UTC m=+0.017965638 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:22:27.728 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:27 smithi153 podman[72972]: 2024-03-28 02:22:27.482815619 +0000 UTC m=+0.148942185 container create 6f30b678733cc5aa72ac7920ae8236a7f40136753708bc71d4c796e0aaabde7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:22:27.728 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:27 smithi153 podman[72972]: 2024-03-28 02:22:27.735455968 +0000 UTC m=+0.401582540 container init 6f30b678733cc5aa72ac7920ae8236a7f40136753708bc71d4c796e0aaabde7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T02:22:27.983 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:27 smithi153 podman[72972]: 2024-03-28 02:22:27.739072857 +0000 UTC m=+0.405199431 container start 6f30b678733cc5aa72ac7920ae8236a7f40136753708bc71d4c796e0aaabde7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T02:22:27.984 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:27 smithi153 podman[72972]: 2024-03-28 02:22:27.835862258 +0000 UTC m=+0.501988837 container attach 6f30b678733cc5aa72ac7920ae8236a7f40136753708bc71d4c796e0aaabde7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T02:22:28.290 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 28 02:22:28 smithi153 podman[72972]: 2024-03-28 02:22:28.029057496 +0000 UTC m=+0.695184068 container died 6f30b678733cc5aa72ac7920ae8236a7f40136753708bc71d4c796e0aaabde7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:22:28.623 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.1.service' 2024-03-28T02:22:28.665 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:28.665 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-28T02:22:28.665 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-28T02:22:28.665 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.2 2024-03-28T02:22:28.970 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:28 smithi178 systemd[1]: Stopping Ceph osd.2 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:29.372 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:28 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2[41645]: 2024-03-28T02:22:28.975+0000 7fdbef0db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:29.373 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:28 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2[41645]: 2024-03-28T02:22:28.975+0000 7fdbef0db700 -1 osd.2 106 *** Got signal Terminated *** 2024-03-28T02:22:29.373 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:28 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2[41645]: 2024-03-28T02:22:28.975+0000 7fdbef0db700 -1 osd.2 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:22:34.704 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:34 smithi178 podman[73253]: 2024-03-28 02:22:34.39797451 +0000 UTC m=+5.532454776 container died 7f2f6d500e61c38869d88ed5860bc630950dc925eac49ff51e8dc78428a67d57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T02:22:35.065 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:34 smithi178 podman[73253]: 2024-03-28 02:22:34.81841211 +0000 UTC m=+5.952892375 container cleanup 7f2f6d500e61c38869d88ed5860bc630950dc925eac49ff51e8dc78428a67d57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:22:35.065 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:34 smithi178 bash[73253]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2 2024-03-28T02:22:35.372 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:35 smithi178 podman[73270]: 2024-03-28 02:22:35.072202304 +0000 UTC m=+0.669193783 container remove 7f2f6d500e61c38869d88ed5860bc630950dc925eac49ff51e8dc78428a67d57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True) 2024-03-28T02:22:35.777 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:35 smithi178 podman[73359]: 2024-03-28 02:22:35.378647222 +0000 UTC m=+0.017582217 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:22:35.777 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:35 smithi178 podman[73359]: 2024-03-28 02:22:35.513159126 +0000 UTC m=+0.152094138 container create 60fbada51f1ea1fa63fb4936750727390fdd773f541038cfd6a7ecc22ec232aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T02:22:36.070 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:35 smithi178 podman[73359]: 2024-03-28 02:22:35.783649765 +0000 UTC m=+0.422584741 container init 60fbada51f1ea1fa63fb4936750727390fdd773f541038cfd6a7ecc22ec232aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:22:36.070 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:35 smithi178 podman[73359]: 2024-03-28 02:22:35.787247739 +0000 UTC m=+0.426182718 container start 60fbada51f1ea1fa63fb4936750727390fdd773f541038cfd6a7ecc22ec232aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-28T02:22:36.070 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:35 smithi178 podman[73359]: 2024-03-28 02:22:35.891318357 +0000 UTC m=+0.530253336 container attach 60fbada51f1ea1fa63fb4936750727390fdd773f541038cfd6a7ecc22ec232aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-28T02:22:36.335 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 28 02:22:36 smithi178 podman[73359]: 2024-03-28 02:22:36.076935278 +0000 UTC m=+0.715870260 container died 60fbada51f1ea1fa63fb4936750727390fdd773f541038cfd6a7ecc22ec232aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T02:22:36.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.2.service' 2024-03-28T02:22:36.760 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:36.761 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-28T02:22:36.761 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-28T02:22:36.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.3 2024-03-28T02:22:37.051 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:36 smithi106 systemd[1]: Stopping Ceph osd.3 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:37.333 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3[53254]: 2024-03-28T02:22:37.079+0000 7ffb1cd68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:37.334 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3[53254]: 2024-03-28T02:22:37.079+0000 7ffb1cd68700 -1 osd.3 106 *** Got signal Terminated *** 2024-03-28T02:22:37.334 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:37 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3[53254]: 2024-03-28T02:22:37.079+0000 7ffb1cd68700 -1 osd.3 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:22:39.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:39 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:39.271+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:39.833 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:39 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:39.570+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:40.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:40 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:40.314+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:40.833 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:40 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:40.581+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:41.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:41.311+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:41.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:41.352+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:41.984 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:41 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:41.594+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:42.083 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:41 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:41.631+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:42.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:42 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:42.356+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:42.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:42 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:42.396+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:42.665 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:42 smithi106 podman[93422]: 2024-03-28 02:22:42.412277674 +0000 UTC m=+5.454157752 container died 7eada527844e43b1ec2db388db73291693688d0395fd25053f4d2084d98e67b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T02:22:42.983 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:42 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:42.586+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:43.033 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:42 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:42.672+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:43.033 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:42 smithi106 podman[93422]: 2024-03-28 02:22:42.780177565 +0000 UTC m=+5.822057651 container cleanup 7eada527844e43b1ec2db388db73291693688d0395fd25053f4d2084d98e67b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T02:22:43.033 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:42 smithi106 bash[93422]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3 2024-03-28T02:22:43.333 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:43 smithi106 podman[93451]: 2024-03-28 02:22:43.039775407 +0000 UTC m=+0.625028484 container remove 7eada527844e43b1ec2db388db73291693688d0395fd25053f4d2084d98e67b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T02:22:43.594 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:43 smithi106 podman[93539]: 2024-03-28 02:22:43.247719599 +0000 UTC m=+0.017899129 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:22:43.595 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:43 smithi106 podman[93539]: 2024-03-28 02:22:43.357244562 +0000 UTC m=+0.127424092 container create e1ea932bcc961d6a0e3f5e681b4df58e6d910df7d8317b3b3e56cf5bf1ad7ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T02:22:43.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:43 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:43.404+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:43.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:43 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:43.352+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:43.892 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:43 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:43.707+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:43.892 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:43 smithi106 podman[93539]: 2024-03-28 02:22:43.601781758 +0000 UTC m=+0.371961288 container init e1ea932bcc961d6a0e3f5e681b4df58e6d910df7d8317b3b3e56cf5bf1ad7ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T02:22:43.892 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:43 smithi106 podman[93539]: 2024-03-28 02:22:43.604979716 +0000 UTC m=+0.375159254 container start e1ea932bcc961d6a0e3f5e681b4df58e6d910df7d8317b3b3e56cf5bf1ad7ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:22:43.892 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:43 smithi106 podman[93539]: 2024-03-28 02:22:43.693347556 +0000 UTC m=+0.463527094 container attach e1ea932bcc961d6a0e3f5e681b4df58e6d910df7d8317b3b3e56cf5bf1ad7ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3-deactivate, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:22:43.983 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:43 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:43.589+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:44.206 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 28 02:22:43 smithi106 podman[93539]: 2024-03-28 02:22:43.898763951 +0000 UTC m=+0.668943488 container died e1ea932bcc961d6a0e3f5e681b4df58e6d910df7d8317b3b3e56cf5bf1ad7ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:22:44.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.3.service' 2024-03-28T02:22:44.558 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:44.558 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-28T02:22:44.558 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-28T02:22:44.558 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.4 2024-03-28T02:22:44.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:44.392+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:44.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:44.370+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:44.833 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:44 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:44.689+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:44.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:44 smithi153 systemd[1]: Stopping Ceph osd.4 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:44.871 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:44 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:44.585+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:45.234 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:44 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4[47346]: 2024-03-28T02:22:44.876+0000 7ff6868f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:45.234 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:44 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4[47346]: 2024-03-28T02:22:44.876+0000 7ff6868f6700 -1 osd.4 106 *** Got signal Terminated *** 2024-03-28T02:22:45.234 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:44 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4[47346]: 2024-03-28T02:22:44.876+0000 7ff6868f6700 -1 osd.4 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:22:45.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:45.366+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:45.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:45.353+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:45.984 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:45 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:45.548+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:46.083 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:45 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:45.684+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:46.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:46 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:46.367+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:46.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:46 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:46.397+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:46.984 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:46 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:46.567+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:47.083 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:46 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:46.728+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:47.484 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:47 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4[47346]: 2024-03-28T02:22:47.031+0000 7ff67eee2700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.323490+0000 front 2024-03-28T02:22:22.323525+0000 (oldest deadline 2024-03-28T02:22:46.423658+0000) 2024-03-28T02:22:47.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:47 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:47.364+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:47.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:47 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:47.365+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:47.984 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:47 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:47.549+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:48.083 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:47 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:47.718+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:48.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:47 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:47.718+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:48.484 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:48 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4[47346]: 2024-03-28T02:22:48.045+0000 7ff67eee2700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.323490+0000 front 2024-03-28T02:22:22.323525+0000 (oldest deadline 2024-03-28T02:22:46.423658+0000) 2024-03-28T02:22:48.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:48 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:48.352+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:48.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:48 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:48.338+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:48.984 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:48 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:48.540+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:49.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:48 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:48.732+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:49.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:48 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:48.732+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:49.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:48 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:48.675+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:49.484 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:49 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4[47346]: 2024-03-28T02:22:49.011+0000 7ff67eee2700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.323490+0000 front 2024-03-28T02:22:22.323525+0000 (oldest deadline 2024-03-28T02:22:46.423658+0000) 2024-03-28T02:22:49.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:49 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:49.139+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:49.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:49 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:49.371+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:49.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:49 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:49.359+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:49.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:49 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:49.359+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:23.314859+0000 front 2024-03-28T02:22:23.314642+0000 (oldest deadline 2024-03-28T02:22:49.214681+0000) 2024-03-28T02:22:49.984 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:49 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:49.512+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:49.984 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:49 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:49.565+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:50.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:49 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:49.733+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:50.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:49 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:49.733+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:50.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:49 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:49.699+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:50.377 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:50 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:50.332+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:50.378 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:50 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:50.332+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:23.314859+0000 front 2024-03-28T02:22:23.314642+0000 (oldest deadline 2024-03-28T02:22:49.214681+0000) 2024-03-28T02:22:50.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:50 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:50.178+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:50.643 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:50 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:50.380+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:50.687 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:50 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:50.559+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:50.688 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:50 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:50.484+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:50.688 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:50 smithi153 podman[73092]: 2024-03-28 02:22:50.422945218 +0000 UTC m=+5.658010947 container died bd17c14eea9aa0fba161e0fd4f9094af7b8054444a05ff20a2baea63267cbc36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:22:50.984 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:50 smithi153 podman[73092]: 2024-03-28 02:22:50.792911164 +0000 UTC m=+6.027976887 container cleanup bd17c14eea9aa0fba161e0fd4f9094af7b8054444a05ff20a2baea63267cbc36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T02:22:50.984 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:50 smithi153 bash[73092]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4 2024-03-28T02:22:51.083 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:50 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:50.764+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:51.083 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:50 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:50.764+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:51.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:50 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:50.649+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:51.441 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:51 smithi153 podman[73109]: 2024-03-28 02:22:51.094629716 +0000 UTC m=+0.666764205 container remove bd17c14eea9aa0fba161e0fd4f9094af7b8054444a05ff20a2baea63267cbc36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:22:51.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:51 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:51.158+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:51.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:51 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:51.392+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:51.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:51 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:51.318+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:51.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:51 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:51.318+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:23.314859+0000 front 2024-03-28T02:22:23.314642+0000 (oldest deadline 2024-03-28T02:22:49.214681+0000) 2024-03-28T02:22:51.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:51 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:51.588+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:51.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:51 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:51.447+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:51.735 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:51 smithi153 podman[73200]: 2024-03-28 02:22:51.351532475 +0000 UTC m=+0.017388986 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:22:51.735 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:51 smithi153 podman[73200]: 2024-03-28 02:22:51.470460589 +0000 UTC m=+0.136317142 container create 40df9071b5bdb074b12397355db61b1bb1a499ee660c86a1a3ff0b3b68c78388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T02:22:51.735 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:51 smithi153 podman[73200]: 2024-03-28 02:22:51.664768982 +0000 UTC m=+0.330625472 container init 40df9071b5bdb074b12397355db61b1bb1a499ee660c86a1a3ff0b3b68c78388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True) 2024-03-28T02:22:51.735 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:51 smithi153 podman[73200]: 2024-03-28 02:22:51.667767189 +0000 UTC m=+0.333623696 container start 40df9071b5bdb074b12397355db61b1bb1a499ee660c86a1a3ff0b3b68c78388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T02:22:52.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:51 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:51.745+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:52.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:51 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:51.745+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:52.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:51 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:51.647+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:52.234 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:51 smithi153 podman[73200]: 2024-03-28 02:22:51.764525251 +0000 UTC m=+0.430381749 container attach 40df9071b5bdb074b12397355db61b1bb1a499ee660c86a1a3ff0b3b68c78388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T02:22:52.234 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 28 02:22:51 smithi153 podman[73200]: 2024-03-28 02:22:51.955646252 +0000 UTC m=+0.621502759 container died 40df9071b5bdb074b12397355db61b1bb1a499ee660c86a1a3ff0b3b68c78388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T02:22:52.545 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:52 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:52.407+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:52.545 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:52 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:52.407+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:22:52.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.4.service' 2024-03-28T02:22:52.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:52 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:52.116+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:52.607 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:52 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:52.306+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:52.607 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:52 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:52.306+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:23.314859+0000 front 2024-03-28T02:22:23.314642+0000 (oldest deadline 2024-03-28T02:22:49.214681+0000) 2024-03-28T02:22:52.607 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:52 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:52.382+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:52.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:22:52.608 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-28T02:22:52.608 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-28T02:22:52.608 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.5 2024-03-28T02:22:52.873 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:52 smithi178 systemd[1]: Stopping Ceph osd.5 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:22:52.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:52 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:52.613+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:52.984 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:52 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:52.630+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:53.076 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:52 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:52.699+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:53.076 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:52 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:52.699+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:53.303 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:52 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:52.931+0000 7fa0bc72a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:22:53.303 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:52 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:52.931+0000 7fa0bc72a700 -1 osd.5 106 *** Got signal Terminated *** 2024-03-28T02:22:53.303 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:52 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:52.931+0000 7fa0bc72a700 -1 osd.5 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:22:53.303 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:53 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:53.307+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:53.303 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:53 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:53.307+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:23.314859+0000 front 2024-03-28T02:22:23.314642+0000 (oldest deadline 2024-03-28T02:22:49.214681+0000) 2024-03-28T02:22:53.333 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:53 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:53.082+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:53.593 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:53 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:53.357+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:53.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:53 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:53.633+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:53.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:53 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:53.431+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:53.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:53 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:53.431+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:22:53.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:53 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:53.598+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:54.083 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:53 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:53.705+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:54.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:53 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:53.705+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:54.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:54 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:54.108+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:54.607 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:54.389+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:54.607 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:54.283+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:54.607 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:54.283+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:23.314859+0000 front 2024-03-28T02:22:23.314642+0000 (oldest deadline 2024-03-28T02:22:49.214681+0000) 2024-03-28T02:22:54.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:54 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:54.677+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:54.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:54 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:54.467+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:54.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:54 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:54.467+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:22:54.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:54 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:54.611+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:55.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:54 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:54.721+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:55.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:54 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:54.721+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:55.583 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:55 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:55.150+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:55.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:55 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:55.363+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:55.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:55 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:55.299+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:55.623 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:55 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:55.299+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:23.314859+0000 front 2024-03-28T02:22:23.314642+0000 (oldest deadline 2024-03-28T02:22:49.214681+0000) 2024-03-28T02:22:55.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:55 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:55.654+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:55.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:55 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:55.464+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:55.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:55 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:55.464+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:22:56.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:55 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:55.743+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:56.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:55 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:55.743+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:56.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:55 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:55.635+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:56.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:56 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:56.169+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:56.598 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:56.336+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:56.599 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:56.326+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:56.599 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:56.326+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:23.314859+0000 front 2024-03-28T02:22:23.314642+0000 (oldest deadline 2024-03-28T02:22:49.214681+0000) 2024-03-28T02:22:56.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:56 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:56.628+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:56.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:56 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:56.432+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:56.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:56 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:56.432+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:22:56.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:56.603+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:56.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:56 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:56.603+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:22:57.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:56 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:56.722+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:57.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:56 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:56.722+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:57.581 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:57 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:57.340+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.113616+0000 front 2024-03-28T02:22:15.113561+0000 (oldest deadline 2024-03-28T02:22:39.213564+0000) 2024-03-28T02:22:57.581 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:57 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5[48614]: 2024-03-28T02:22:57.340+0000 7fa0b4d16700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:23.314859+0000 front 2024-03-28T02:22:23.314642+0000 (oldest deadline 2024-03-28T02:22:49.214681+0000) 2024-03-28T02:22:57.582 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:57 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:57.312+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:57.582 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:57 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:57.312+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:22:57.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:57 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:57.149+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:57.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:57 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:57.149+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:22:57.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:57.398+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:57.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:57.398+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:22:57.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:57.614+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:57.735 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:57 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:57.614+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:22:57.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:57 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:57.586+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:57.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:57 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:57.586+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:22:58.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:57 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:57.680+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:58.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:57 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:57.680+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:58.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:58 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:58.153+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:58.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:58 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:58.153+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:22:58.621 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:58 smithi178 podman[73490]: 2024-03-28 02:22:58.451157359 +0000 UTC m=+5.644721138 container died 785f9c0b6b34bad8bc20cbffc3d5726cc5baf7210aeae7e3b733288c71601349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T02:22:58.621 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:58 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:58.323+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:58.621 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:58 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:58.323+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:22:58.639 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:58 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:58.368+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:58.639 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:58 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:58.368+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:22:58.639 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:58 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:58.368+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:22:58.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:58 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:58.626+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:58.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:58 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:58.626+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:22:58.874 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:58 smithi178 podman[73490]: 2024-03-28 02:22:58.848983558 +0000 UTC m=+6.042547314 container cleanup 785f9c0b6b34bad8bc20cbffc3d5726cc5baf7210aeae7e3b733288c71601349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T02:22:58.874 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:58 smithi178 bash[73490]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5 2024-03-28T02:22:58.984 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:58 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:58.644+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:58.984 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:58 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:58.644+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:22:59.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:58 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:58.683+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:22:59.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:58 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:58.683+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:22:59.373 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:59 smithi178 podman[73506]: 2024-03-28 02:22:59.058972979 +0000 UTC m=+0.605229453 container remove 785f9c0b6b34bad8bc20cbffc3d5726cc5baf7210aeae7e3b733288c71601349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T02:22:59.373 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:59 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:59.276+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:22:59.373 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:22:59 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:22:59.276+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:22:59.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:59 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:59.164+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:22:59.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:22:59 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:22:59.164+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:22:59.598 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:59 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:59.350+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:22:59.598 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:59 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:59.350+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:22:59.598 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:22:59 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:22:59.350+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:22:59.707 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:59 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:59.588+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:22:59.707 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:22:59 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:22:59.588+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:22:59.707 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:59 smithi178 podman[73599]: 2024-03-28 02:22:59.355661815 +0000 UTC m=+0.017766698 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:22:59.707 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:59 smithi178 podman[73599]: 2024-03-28 02:22:59.468395936 +0000 UTC m=+0.130500829 container create 515ad06bf1d3ef077c40cde91fdd35bc857adbff7953afc8fb61112774aee803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T02:22:59.984 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:59 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:59.603+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:22:59.984 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:22:59 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:22:59.603+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:00.003 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:59 smithi178 podman[73599]: 2024-03-28 02:22:59.712883918 +0000 UTC m=+0.374988795 container init 515ad06bf1d3ef077c40cde91fdd35bc857adbff7953afc8fb61112774aee803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T02:23:00.003 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:59 smithi178 podman[73599]: 2024-03-28 02:22:59.716260091 +0000 UTC m=+0.378364965 container start 515ad06bf1d3ef077c40cde91fdd35bc857adbff7953afc8fb61112774aee803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-28T02:23:00.003 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:22:59 smithi178 podman[73599]: 2024-03-28 02:22:59.804471728 +0000 UTC m=+0.466576612 container attach 515ad06bf1d3ef077c40cde91fdd35bc857adbff7953afc8fb61112774aee803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:23:00.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:59 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:59.677+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:00.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:59 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:59.677+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:00.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:22:59 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:22:59.677+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:00.302 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 28 02:23:00 smithi178 podman[73599]: 2024-03-28 02:23:00.009048741 +0000 UTC m=+0.671153640 container died 515ad06bf1d3ef077c40cde91fdd35bc857adbff7953afc8fb61112774aee803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-28T02:23:00.302 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:00.249+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:00.302 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:00.249+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:00.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:00.211+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:23:00.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:00.211+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:23:00.602 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:00.542+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:00.603 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:00 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:00.542+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:00.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.5.service' 2024-03-28T02:23:00.679 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:23:00.679 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-28T02:23:00.679 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-28T02:23:00.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.6 2024-03-28T02:23:00.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:00.555+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:00.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:00.555+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:00.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:00.388+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:00.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:00.388+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:00.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:00 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:00.388+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:01.084 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:00 smithi106 systemd[1]: Stopping Ceph osd.6 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:23:01.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:00.685+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:01.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:00.685+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:01.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:00 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:00.685+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:01.563 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:01 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:01.240+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:01.563 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:01 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:01.240+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:01.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:01.198+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:23:01.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:01.198+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:23:01.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:01.266+0000 7f55d3542700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:23:01.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:01.266+0000 7f55d3542700 -1 osd.6 106 *** Got signal Terminated *** 2024-03-28T02:23:01.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:01.266+0000 7f55d3542700 -1 osd.6 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:23:01.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:01 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:01.554+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:01.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:01 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:01.554+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:01.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:01 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:01.347+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:01.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:01 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:01.347+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:01.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:01 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:01.347+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:01.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:01 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:01.568+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:01.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:01 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:01.568+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:02.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:01.650+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:02.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:01.650+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:02.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:01.650+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:02.572 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:02 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:02.279+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:02.572 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:02 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:02.279+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:02.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:02 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:02.189+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:23:02.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:02 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:02.189+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:23:02.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:02 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:02.512+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:02.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:02 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:02.512+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:02.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:02 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:02.311+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:02.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:02 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:02.311+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:02.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:02 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:02.311+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:02.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:02 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:02.577+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:02.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:02 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:02.577+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:03.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:02 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:02.612+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:03.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:02 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:02.612+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:03.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:02 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:02.612+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:03.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:03 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:03.143+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:23:03.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:03 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:03.143+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:23:03.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:03 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:03.143+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.538608+0000 front 2024-03-28T02:22:36.538661+0000 (oldest deadline 2024-03-28T02:23:02.438760+0000) 2024-03-28T02:23:03.604 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:03 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:03.286+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:03.604 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:03 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:03.286+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:03.604 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:03 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:03.286+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:03.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:03 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:03.506+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:03.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:03 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:03.506+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:03.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:03 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:03.506+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:03.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:03 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:03.270+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:03.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:03 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:03.270+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:03.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:03 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:03.270+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:03.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:03 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:03.609+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:03.873 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:03 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:03.609+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:04.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:03 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:03.655+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:04.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:03 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:03.655+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:04.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:03 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:03.655+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:04.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:04 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:04.138+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:23:04.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:04 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:04.138+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:23:04.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:04 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:04.138+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.538608+0000 front 2024-03-28T02:22:36.538661+0000 (oldest deadline 2024-03-28T02:23:02.438760+0000) 2024-03-28T02:23:04.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:04 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:04.273+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:04.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:04 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:04.273+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:04.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:04 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:04.273+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:04.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:04 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:04.502+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:04.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:04 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:04.502+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:04.735 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:04 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:04.502+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:04.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:04 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:04.312+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:04.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:04 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:04.312+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:04.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:04 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:04.312+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:04.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:04 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:04.312+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:05.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:04 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:04.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:05.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:04 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:04.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:05.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:04 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:04.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:05.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:04 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:04.644+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:05.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:04 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:04.644+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:05.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:05 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:05.119+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:23:05.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:05 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:05.119+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:23:05.584 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:05 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:05.119+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.538608+0000 front 2024-03-28T02:22:36.538661+0000 (oldest deadline 2024-03-28T02:23:02.438760+0000) 2024-03-28T02:23:05.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:05 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:05.267+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:05.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:05 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:05.267+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:05.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:05 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:05.267+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:05.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:05 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:05.501+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:05.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:05 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:05.501+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:05.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:05 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:05.501+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:05.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:05 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:05.334+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:05.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:05 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:05.334+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:05.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:05 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:05.334+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:05.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:05 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:05.334+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:06.078 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:05 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:05.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:06.079 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:05 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:05.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:06.079 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:05 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:05.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:06.079 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:05 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:05.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:06.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:05 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:05.650+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:06.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:05 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:05.650+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:06.334 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:06 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:06.084+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.837003+0000 front 2024-03-28T02:22:24.837091+0000 (oldest deadline 2024-03-28T02:22:48.337028+0000) 2024-03-28T02:23:06.334 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:06 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:06.084+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:31.238250+0000 front 2024-03-28T02:22:31.238265+0000 (oldest deadline 2024-03-28T02:22:56.538135+0000) 2024-03-28T02:23:06.334 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:06 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6[60105]: 2024-03-28T02:23:06.084+0000 7f55cfb36700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.538608+0000 front 2024-03-28T02:22:36.538661+0000 (oldest deadline 2024-03-28T02:23:02.438760+0000) 2024-03-28T02:23:06.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:06 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:06.295+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:06.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:06 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:06.295+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:06.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:06 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:06.295+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:06.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:06 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:06.471+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:06.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:06 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:06.471+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:06.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:06 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:06.471+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:06.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:06 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:06.327+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:06.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:06 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:06.327+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:06.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:06 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:06.327+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:06.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:06 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:06.327+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:06.903 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:06 smithi106 podman[93643]: 2024-03-28 02:23:06.604291345 +0000 UTC m=+5.725096291 container died 47e2d6065bd1b4be7ccaef8dea6bbbe0b5bfc263edd624ad9f16790367fa5246 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T02:23:06.903 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:06 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:06.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:06.903 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:06 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:06.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:06.903 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:06 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:06.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:06.904 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:06 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:06.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:07.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:06 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:06.647+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:07.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:06 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:06.647+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:07.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:06 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:06.647+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:07.245 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:07 smithi106 podman[93643]: 2024-03-28 02:23:07.00890143 +0000 UTC m=+6.129706382 container cleanup 47e2d6065bd1b4be7ccaef8dea6bbbe0b5bfc263edd624ad9f16790367fa5246 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T02:23:07.245 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:07 smithi106 bash[93643]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6 2024-03-28T02:23:07.562 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:07 smithi106 podman[93660]: 2024-03-28 02:23:07.25211313 +0000 UTC m=+0.649278158 container remove 47e2d6065bd1b4be7ccaef8dea6bbbe0b5bfc263edd624ad9f16790367fa5246 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T02:23:07.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:07 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:07.314+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:07.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:07 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:07.314+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:07.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:07 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:07.314+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:07.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:07 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:07.431+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:07.735 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:07 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:07.431+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:07.735 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:07 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:07.431+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:07.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:07 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:07.295+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:07.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:07 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:07.295+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:07.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:07 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:07.295+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:07.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:07 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:07.295+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:07.834 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:07 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:07.639+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:07.834 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:07 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:07.639+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:07.834 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:07 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:07.639+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:07.834 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:07 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:07.639+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:07.834 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:07 smithi106 podman[93754]: 2024-03-28 02:23:07.569460873 +0000 UTC m=+0.112882212 container create 611deb7883ab000d3c559d67c139cf72d9d9ecf95bd03be53b8a0cdfc388dfd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:23:07.835 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:07 smithi106 podman[93754]: 2024-03-28 02:23:07.474297372 +0000 UTC m=+0.017718719 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:23:07.835 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:07 smithi106 podman[93754]: 2024-03-28 02:23:07.788880234 +0000 UTC m=+0.332301572 container init 611deb7883ab000d3c559d67c139cf72d9d9ecf95bd03be53b8a0cdfc388dfd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T02:23:07.835 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:07 smithi106 podman[93754]: 2024-03-28 02:23:07.791969945 +0000 UTC m=+0.335391284 container start 611deb7883ab000d3c559d67c139cf72d9d9ecf95bd03be53b8a0cdfc388dfd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T02:23:08.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:07 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:07.653+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:08.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:07 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:07.653+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:08.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:07 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:07.653+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:08.334 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:07 smithi106 podman[93754]: 2024-03-28 02:23:07.867633711 +0000 UTC m=+0.411055076 container attach 611deb7883ab000d3c559d67c139cf72d9d9ecf95bd03be53b8a0cdfc388dfd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:23:08.334 INFO:journalctl@ceph.osd.6.smithi106.stdout:Mar 28 02:23:08 smithi106 podman[93754]: 2024-03-28 02:23:08.077813163 +0000 UTC m=+0.621234503 container died 611deb7883ab000d3c559d67c139cf72d9d9ecf95bd03be53b8a0cdfc388dfd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:23:08.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:08 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:08.270+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:08.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:08 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:08.270+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:08.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:08 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:08.270+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:08.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.6.service' 2024-03-28T02:23:08.730 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:23:08.730 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-28T02:23:08.730 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-28T02:23:08.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.7 2024-03-28T02:23:08.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:08 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:08.419+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:08.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:08 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:08.419+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:08.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:08 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:08.419+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:08.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:08 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:08.276+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:08.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:08 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:08.276+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:08.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:08 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:08.276+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:08.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:08 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:08.276+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:09.040 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:08 smithi153 systemd[1]: Stopping Ceph osd.7 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:23:09.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:08 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:08.638+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:09.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:08 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:08.638+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:09.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:08 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:08.638+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:09.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:08 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:08.639+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:09.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:08 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:08.646+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:09.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:08 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:08.646+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:09.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:08 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:08.646+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:09.302 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:09.045+0000 7f94dbd5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:23:09.302 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:09.045+0000 7f94dbd5b700 -1 osd.7 106 *** Got signal Terminated *** 2024-03-28T02:23:09.302 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:09.045+0000 7f94dbd5b700 -1 osd.7 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:23:09.303 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:09.252+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:09.303 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:09.252+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:09.303 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:09.252+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:09.303 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:09.252+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:09.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:09 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:09.225+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:09.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:09 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:09.225+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:09.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:09 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:09.225+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:09.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:09.457+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:09.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:09.457+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:09.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:09.457+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:10.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:09.595+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:10.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:09.595+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:10.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:09.595+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:10.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:09 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:09.595+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:10.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:09 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:09.692+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:10.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:09 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:09.692+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:10.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:09 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:09.692+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:10.484 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:10.460+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:10.484 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:10.460+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:10.484 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:10.460+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:10.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:10.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:10.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:10.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:10.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:10.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:10.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:10 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:10.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:10.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:10 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:10.254+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:10.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:10 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:10.254+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:10.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:10 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:10.254+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:11.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:10.644+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:11.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:10.644+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:11.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:10.644+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:11.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:10 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:10.644+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:11.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:10 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:10.739+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:11.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:10 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:10.739+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:11.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:10 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:10.739+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:11.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:10 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:10.739+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:11.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:11.238+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:11.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:11.238+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:11.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:11.238+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:11.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:11.238+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:11.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:11.220+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:11.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:11.220+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:11.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:11.220+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:11.984 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:11.505+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:11.984 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:11.505+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:11.984 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:11 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:11.505+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:12.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:11.667+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:12.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:11.667+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:12.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:11.667+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:12.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:11 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:11.667+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:12.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:11.740+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:12.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:11.740+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:12.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:11.740+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:12.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:11 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:11.740+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:12.467 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:12.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:12.467 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:12.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:12.467 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:12.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:12.467 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:12.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:12.467 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:12.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:12.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:12.183+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:12.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:12.183+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:12.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:12.183+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:12.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:12.472+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:12.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:12.472+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:12.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:12 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:12.472+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:13.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:12.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:13.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:12.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:13.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:12.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:13.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:12 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:12.669+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:13.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:12.707+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:13.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:12.707+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:13.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:12.707+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:13.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:12 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:12.707+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:13.469 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:13.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:13.469 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:13.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:13.470 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:13.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:13.470 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:13.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:13.470 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:13.214+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:13.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:13.199+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:13.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:13.199+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:13.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:13.199+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:13.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:13.199+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:13.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:13.474+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:24.169503+0000 front 2024-03-28T02:22:24.169479+0000 (oldest deadline 2024-03-28T02:22:48.770375+0000) 2024-03-28T02:23:13.734 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:13.474+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.972142+0000 front 2024-03-28T02:22:33.972004+0000 (oldest deadline 2024-03-28T02:22:56.871744+0000) 2024-03-28T02:23:13.735 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7[54369]: 2024-03-28T02:23:13.474+0000 7f94d834f700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:36.872250+0000 front 2024-03-28T02:22:36.872355+0000 (oldest deadline 2024-03-28T02:23:03.272950+0000) 2024-03-28T02:23:14.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:13.664+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:14.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:13.664+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:14.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:13.664+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:14.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:13 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:13.664+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:14.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:13.686+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:14.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:13.686+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:14.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:13.686+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:14.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:13 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:13.686+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:14.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:14 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:14.166+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:14.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:14 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:14.166+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:14.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:14 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:14.166+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:14.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:14 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:14.166+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:14.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:14 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:14.166+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:14.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:14.206+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:14.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:14.206+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:14.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:14.206+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:14.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:14.206+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:14.806 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:14 smithi153 podman[73303]: 2024-03-28 02:23:14.556021517 +0000 UTC m=+5.617328831 container died c5da74a232df2b130104c91972ec01d83947ff5d1828b88f5c4f6a78a686a4ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T02:23:15.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:14.644+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:15.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:14.644+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:15.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:14.644+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:15.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:14.644+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:15.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:14 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:14.644+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:15.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:14.717+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:15.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:14.717+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:15.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:14.717+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:15.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:14 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:14.717+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:15.189 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:14 smithi153 podman[73303]: 2024-03-28 02:23:14.920454451 +0000 UTC m=+5.981761755 container cleanup c5da74a232df2b130104c91972ec01d83947ff5d1828b88f5c4f6a78a686a4ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:23:15.189 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:14 smithi153 bash[73303]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7 2024-03-28T02:23:15.484 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:15 smithi153 podman[73332]: 2024-03-28 02:23:15.196588585 +0000 UTC m=+0.641033612 container remove c5da74a232df2b130104c91972ec01d83947ff5d1828b88f5c4f6a78a686a4ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:23:15.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:15 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:15.194+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:15.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:15 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:15.194+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:15.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:15 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:15.194+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:15.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:15 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:15.194+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:15.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:15 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:15.194+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:15.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:15.182+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:15.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:15.182+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:15.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:15.182+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:15.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:15.182+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:15.786 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:15 smithi153 podman[73423]: 2024-03-28 02:23:15.433428596 +0000 UTC m=+0.017433186 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:23:15.786 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:15 smithi153 podman[73423]: 2024-03-28 02:23:15.556241699 +0000 UTC m=+0.140246294 container create f3d9bf5e0bc29428cb1a969ff9900a2d5813820c5c99b0b107aaf5af96516a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:23:15.786 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:15 smithi153 podman[73423]: 2024-03-28 02:23:15.792024747 +0000 UTC m=+0.376029326 container init f3d9bf5e0bc29428cb1a969ff9900a2d5813820c5c99b0b107aaf5af96516a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T02:23:16.075 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:15 smithi153 podman[73423]: 2024-03-28 02:23:15.795409596 +0000 UTC m=+0.379414172 container start f3d9bf5e0bc29428cb1a969ff9900a2d5813820c5c99b0b107aaf5af96516a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T02:23:16.076 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:15 smithi153 podman[73423]: 2024-03-28 02:23:15.867070748 +0000 UTC m=+0.451075332 container attach f3d9bf5e0bc29428cb1a969ff9900a2d5813820c5c99b0b107aaf5af96516a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T02:23:16.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:15 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:15.649+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:16.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:15 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:15.649+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:16.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:15 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:15.649+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:16.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:15 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:15.649+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:16.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:15 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:15.649+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:16.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:15.701+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:16.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:15.701+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:16.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:15.701+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:16.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:15 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:15.701+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:16.439 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 28 02:23:16 smithi153 podman[73423]: 2024-03-28 02:23:16.082006648 +0000 UTC m=+0.666011228 container died f3d9bf5e0bc29428cb1a969ff9900a2d5813820c5c99b0b107aaf5af96516a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:23:16.439 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:16 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:16.158+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:16.440 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:16 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:16.158+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:16.440 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:16 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:16.158+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:16.440 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:16 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:16.158+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:16.440 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:16 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:16.158+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:16.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:16 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:16.194+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:16.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:16 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:16.194+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:16.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:16 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:16.194+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:16.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:16 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:16.194+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:16.747 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.7.service' 2024-03-28T02:23:16.832 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:23:16.833 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-28T02:23:16.833 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-28T02:23:16.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.8 2024-03-28T02:23:17.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:16.633+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:17.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:16.633+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:17.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:16.633+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:17.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:16.633+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:17.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:16 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:16.633+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:17.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:16 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:16.723+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:17.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:16 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:16.723+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:17.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:16 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:16.723+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:17.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:16 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:16.723+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:17.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:16 smithi178 systemd[1]: Stopping Ceph osd.8 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:23:17.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:17 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:17.187+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:17.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:17 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:17.187+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:17.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:17 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:17.187+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:17.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:17 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:17.187+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:17.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:17 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:17.187+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:17.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:17.177+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:17.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:17.177+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:17.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:17.177+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:17.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:17.177+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:17.624 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:17.161+0000 7f6574285700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:23:17.624 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:17.161+0000 7f6574285700 -1 osd.8 106 *** Got signal Terminated *** 2024-03-28T02:23:17.624 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:17.161+0000 7f6574285700 -1 osd.8 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:23:18.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:17 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:17.675+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:18.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:17 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:17.675+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:18.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:17 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:17.675+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:18.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:17 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:17.675+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:18.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:17 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:17.675+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:18.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:17.740+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:18.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:17.740+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:18.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:17.740+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:18.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:17 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:17.740+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:18.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:18 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:18.167+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:18.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:18 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:18.167+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:18.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:18 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:18.167+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:18.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:18 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:18.167+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:18.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:18 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:18.167+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:18.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:18 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:18.223+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:18.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:18 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:18.223+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:18.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:18 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:18.223+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:18.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:18 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:18.223+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:19.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:18 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:18.685+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:19.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:18 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:18.685+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:19.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:18 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:18.685+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:19.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:18 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:18.685+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:19.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:18 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:18.685+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:19.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:18 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:18.712+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:19.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:18 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:18.712+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:19.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:18 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:18.712+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:19.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:18 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:18.712+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:19.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:19 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:19.147+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:19.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:19 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:19.147+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:19.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:19 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:19.147+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:19.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:19 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:19.147+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:19.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:19 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:19.147+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:19.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:19.249+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:19.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:19.249+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:19.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:19.249+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:19.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:19.249+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:19.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:19.249+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:20.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:19 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:19.702+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:20.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:19 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:19.702+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:20.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:19 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:19.702+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:20.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:19 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:19.702+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:20.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:19 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:19.702+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:20.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:19 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:19.702+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:20.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:19.745+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:20.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:19.745+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:20.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:19.745+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:20.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:19 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:19.745+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:20.402 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:20 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:20.268+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:20.402 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:20 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:20.268+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:20.402 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:20 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:20.268+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:20.402 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:20 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:20.268+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:20.402 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:20 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:20.268+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:20.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:20 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:20.144+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:20.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:20 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:20.144+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:20.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:20 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:20.144+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:20.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:20 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:20.144+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:20.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:20 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:20.144+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:21.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:20.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:21.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:20.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:21.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:20.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:21.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:20.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:21.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:20.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:21.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:20 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:20.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:21.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:20 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:20.763+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:21.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:20 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:20.763+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:21.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:20 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:20.763+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:21.124 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:20 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:20.763+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:21.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:21 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:21.116+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:21.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:21 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:21.116+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:21.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:21 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:21.116+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:21.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:21 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:21.116+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:21.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:21 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:21.116+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:21.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:21 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:21.116+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:21.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:21.292+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:21.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:21.292+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:21.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:21.292+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:21.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:21.292+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:21.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:21.292+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:22.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:21 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:21.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:22.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:21 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:21.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:22.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:21 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:21.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:22.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:21 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:21.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:22.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:21 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:21.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:22.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:21 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:21.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:22.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:21.801+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:22.523002+0000 front 2024-03-28T02:22:22.522793+0000 (oldest deadline 2024-03-28T02:22:48.422775+0000) 2024-03-28T02:23:22.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:21.801+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.024181+0000 front 2024-03-28T02:22:33.024270+0000 (oldest deadline 2024-03-28T02:22:56.524287+0000) 2024-03-28T02:23:22.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:21.801+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:41.125839+0000 front 2024-03-28T02:22:41.125708+0000 (oldest deadline 2024-03-28T02:23:06.425580+0000) 2024-03-28T02:23:22.123 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8[55207]: 2024-03-28T02:23:21.801+0000 7f6570879700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.426293+0000 front 2024-03-28T02:22:46.426133+0000 (oldest deadline 2024-03-28T02:23:10.526210+0000) 2024-03-28T02:23:22.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:22 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:22.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:22.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:22 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:22.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:22.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:22 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:22.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:22.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:22 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:22.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:22.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:22 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:22.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:22.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:22 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:22.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:22.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:22 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:22.315+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:22.623 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:22 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:22.315+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:22.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:22 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:22.315+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:22.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:22 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:22.315+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:22.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:22 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:22.315+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:22.949 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:22 smithi178 podman[73703]: 2024-03-28 02:23:22.697337528 +0000 UTC m=+5.667184313 container died 0c3f7c27b304e15ba029144c86ce67e1eb4d7ef32edb130dffa092b42d8fe367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:23:23.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:22 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:22.727+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:23.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:22 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:22.727+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:23.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:22 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:22.727+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:23.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:22 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:22.727+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:23.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:22 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:22.727+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:23.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:22 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:22.727+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:23.302 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:23 smithi178 podman[73703]: 2024-03-28 02:23:23.053234292 +0000 UTC m=+6.023081062 container cleanup 0c3f7c27b304e15ba029144c86ce67e1eb4d7ef32edb130dffa092b42d8fe367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T02:23:23.302 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:23 smithi178 bash[73703]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8 2024-03-28T02:23:23.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:23 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:23.051+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:23.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:23 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:23.051+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:23.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:23 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:23.051+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:23.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:23 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:23.051+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:23.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:23 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:23.051+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:23.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:23 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:23.051+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:23.623 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:23 smithi178 podman[73733]: 2024-03-28 02:23:23.363212066 +0000 UTC m=+0.660980774 container remove 0c3f7c27b304e15ba029144c86ce67e1eb4d7ef32edb130dffa092b42d8fe367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T02:23:23.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:23 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:23.307+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:23.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:23 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:23.307+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:23.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:23 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:23.307+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:23.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:23 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:23.307+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:23.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:23 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:23.307+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:23.994 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:23 smithi178 podman[73827]: 2024-03-28 02:23:23.554580717 +0000 UTC m=+0.018120002 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:23:23.995 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:23 smithi178 podman[73827]: 2024-03-28 02:23:23.688995315 +0000 UTC m=+0.152534584 container create 901a70e6d37d02f713c5f03711687abd01c1df11b80903a698667a0b263f8e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-28T02:23:24.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:23 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:23.681+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:24.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:23 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:23.681+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:24.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:23 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:23.681+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:24.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:23 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:23.681+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:24.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:23 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:23.681+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:24.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:23 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:23.681+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:24.286 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:24 smithi178 podman[73827]: 2024-03-28 02:23:24.000288902 +0000 UTC m=+0.463828197 container init 901a70e6d37d02f713c5f03711687abd01c1df11b80903a698667a0b263f8e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T02:23:24.286 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:24 smithi178 podman[73827]: 2024-03-28 02:23:24.003891352 +0000 UTC m=+0.467430624 container start 901a70e6d37d02f713c5f03711687abd01c1df11b80903a698667a0b263f8e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T02:23:24.286 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:24 smithi178 podman[73827]: 2024-03-28 02:23:24.099986016 +0000 UTC m=+0.563525301 container attach 901a70e6d37d02f713c5f03711687abd01c1df11b80903a698667a0b263f8e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T02:23:24.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:24 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:24.043+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:24.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:24 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:24.043+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:24.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:24 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:24.043+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:24.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:24 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:24.043+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:24.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:24 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:24.043+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:24.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:24 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:24.043+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:24.624 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 28 02:23:24 smithi178 podman[73827]: 2024-03-28 02:23:24.293391122 +0000 UTC m=+0.756930398 container died 901a70e6d37d02f713c5f03711687abd01c1df11b80903a698667a0b263f8e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-28T02:23:24.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:24 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:24.290+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:24.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:24 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:24.290+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:24.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:24 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:24.290+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:24.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:24 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:24.290+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:24.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:24 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:24.290+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:25.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.8.service' 2024-03-28T02:23:25.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:24 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:24.717+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:25.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:24 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:24.717+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:25.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:24 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:24.717+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:25.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:24 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:24.717+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:25.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:24 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:24.717+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:25.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:24 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:24.717+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:25.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:23:25.123 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-28T02:23:25.123 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-28T02:23:25.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.9 2024-03-28T02:23:25.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:25 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:25.242+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:25.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:25 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:25.242+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:25.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:25 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:25.242+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:25.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:25 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:25.242+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:25.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:25 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:25.242+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:25.414 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 systemd[1]: Stopping Ceph osd.9 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:23:25.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:25 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:25.021+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:25.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:25 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:25.021+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:25.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:25 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:25.021+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:25.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:25 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:25.021+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:25.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:25 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:25.021+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:25.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:25 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:25.021+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:25.696 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:25.419+0000 7f45ab189700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:23:25.696 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:25.419+0000 7f45ab189700 -1 osd.9 106 *** Got signal Terminated *** 2024-03-28T02:23:25.696 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:25.419+0000 7f45ab189700 -1 osd.9 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:23:25.697 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:25.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:25.697 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:25.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:25.697 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:25.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:25.697 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:25.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:25.697 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:25.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:25.697 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:25 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:25.701+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:26.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:26 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:26.048+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:26.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:26 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:26.048+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:26.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:26 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:26.048+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:26.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:26 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:26.048+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:26.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:26 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:26.048+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:26.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:26 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:26.048+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:26.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:26 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:26.219+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:26.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:26 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:26.219+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:26.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:26 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:26.219+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:26.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:26 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:26.219+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:26.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:26 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:26.219+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:27.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:26 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:26.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:27.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:26 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:26.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:27.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:26 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:26.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:27.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:26 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:26.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:27.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:26 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:26.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:27.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:26 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:26.692+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:27.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:27 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:27.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:27.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:27 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:27.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:27.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:27 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:27.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:27.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:27 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:27.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:27.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:27 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:27.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:27.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:27 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:27.090+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:27.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:27 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:27.184+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:27.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:27 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:27.184+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:27.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:27 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:27.184+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:27.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:27 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:27.184+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:27.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:27 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:27.184+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:28.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:27 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:27.729+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:28.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:27 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:27.729+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:28.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:27 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:27.729+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:28.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:27 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:27.729+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:28.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:27 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:27.729+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:28.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:27 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:27.729+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:28.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:28 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:28.115+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:28.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:28 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:28.115+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:28.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:28 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:28.115+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:28.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:28 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:28.115+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:28.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:28 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:28.115+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:28.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:28 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:28.115+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:28.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:28 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:28.148+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:28.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:28 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:28.148+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:28.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:28 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:28.148+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:28.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:28 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:28.148+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:28.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:28 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:28.148+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:29.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:28 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:28.694+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:29.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:28 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:28.694+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:29.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:28 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:28.694+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:29.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:28 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:28.694+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:29.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:28 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:28.694+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:29.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:28 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:28.694+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:29.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:29 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:29.136+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:29.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:29 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:29.136+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:29.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:29 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:29.136+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:29.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:29 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:29.136+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:29.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:29 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:29.136+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:29.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:29 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:29.136+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:29.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:29 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:29.131+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:29.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:29 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:29.131+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:29.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:29 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:29.131+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:29.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:29 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:29.131+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:29.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:29 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:29.131+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:30.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:29 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:29.648+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:13.426108+0000 front 2024-03-28T02:22:13.426096+0000 (oldest deadline 2024-03-28T02:22:39.326098+0000) 2024-03-28T02:23:30.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:29 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:29.648+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.127536+0000 front 2024-03-28T02:22:25.127617+0000 (oldest deadline 2024-03-28T02:22:46.827659+0000) 2024-03-28T02:23:30.084 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:29 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:29.648+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.630067+0000 front 2024-03-28T02:22:33.630171+0000 (oldest deadline 2024-03-28T02:22:59.530118+0000) 2024-03-28T02:23:30.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:29 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:29.648+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:39.530407+0000 front 2024-03-28T02:22:39.530466+0000 (oldest deadline 2024-03-28T02:23:04.830570+0000) 2024-03-28T02:23:30.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:29 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:29.648+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:48.331620+0000 front 2024-03-28T02:22:48.331852+0000 (oldest deadline 2024-03-28T02:23:14.231686+0000) 2024-03-28T02:23:30.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:29 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9[67153]: 2024-03-28T02:23:29.649+0000 7f45a777d700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:57.633535+0000 front 2024-03-28T02:22:57.633429+0000 (oldest deadline 2024-03-28T02:23:18.733163+0000) 2024-03-28T02:23:30.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:30 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:30.130+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:30.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:30 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:30.130+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:30.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:30 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:30.130+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:30.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:30 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:30.130+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:30.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:30 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:30.130+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:30.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:30 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:30.130+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:30.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:30 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:30.137+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:30.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:30 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:30.137+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:30.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:30 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:30.137+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:30.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:30 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:30.137+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:30.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:30 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:30.137+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:30.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:30 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:30.137+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:31.180 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:30 smithi106 podman[93870]: 2024-03-28 02:23:30.866155875 +0000 UTC m=+5.547798290 container died 663d3ed319bc4301032e42a68ae4d0645fe84e067df08705fd1a20e29873a351 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T02:23:31.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:31 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:31.091+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:31.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:31 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:31.091+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:31.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:31 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:31.091+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:31.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:31 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:31.091+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:31.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:31 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:31.091+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:31.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:31 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:31.091+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:31.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:31 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:31.102+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:31.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:31 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:31.102+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:31.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:31 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:31.102+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:31.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:31 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:31.102+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:31.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:31 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:31.102+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:31.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:31 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:31.102+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:31.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:31 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:31.102+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.862799+0000 front 2024-03-28T02:23:04.862949+0000 (oldest deadline 2024-03-28T02:23:30.162635+0000) 2024-03-28T02:23:31.584 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:31 smithi106 podman[93870]: 2024-03-28 02:23:31.186422925 +0000 UTC m=+5.868065384 container cleanup 663d3ed319bc4301032e42a68ae4d0645fe84e067df08705fd1a20e29873a351 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T02:23:31.584 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:31 smithi106 bash[93870]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9 2024-03-28T02:23:31.585 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:31 smithi106 podman[93888]: 2024-03-28 02:23:31.379216529 +0000 UTC m=+0.510414825 container remove 663d3ed319bc4301032e42a68ae4d0645fe84e067df08705fd1a20e29873a351 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T02:23:32.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:31 smithi106 podman[93980]: 2024-03-28 02:23:31.704830216 +0000 UTC m=+0.017675181 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:23:32.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:31 smithi106 podman[93980]: 2024-03-28 02:23:31.813283097 +0000 UTC m=+0.126128090 container create 1cb9d823a0cf2f6c97b8553e15c8687145676d589254f2be3308eef0a13982bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:23:32.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:32 smithi106 podman[93980]: 2024-03-28 02:23:32.024162021 +0000 UTC m=+0.337006975 container init 1cb9d823a0cf2f6c97b8553e15c8687145676d589254f2be3308eef0a13982bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T02:23:32.085 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:32 smithi106 podman[93980]: 2024-03-28 02:23:32.027220751 +0000 UTC m=+0.340065705 container start 1cb9d823a0cf2f6c97b8553e15c8687145676d589254f2be3308eef0a13982bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T02:23:32.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:32 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:32.122+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:32.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:32 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:32.122+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:32.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:32 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:32.122+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:32.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:32 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:32.122+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:32.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:32 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:32.122+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:32.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:32 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:32.122+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:32 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:32.109+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:32 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:32.109+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:32 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:32.109+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:32 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:32.109+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:32 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:32.109+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:32 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:32.109+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:32 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:32.109+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.862799+0000 front 2024-03-28T02:23:04.862949+0000 (oldest deadline 2024-03-28T02:23:30.162635+0000) 2024-03-28T02:23:32.584 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:32 smithi106 podman[93980]: 2024-03-28 02:23:32.099011123 +0000 UTC m=+0.411856089 container attach 1cb9d823a0cf2f6c97b8553e15c8687145676d589254f2be3308eef0a13982bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T02:23:32.584 INFO:journalctl@ceph.osd.9.smithi106.stdout:Mar 28 02:23:32 smithi106 podman[93980]: 2024-03-28 02:23:32.317277257 +0000 UTC m=+0.630122213 container died 1cb9d823a0cf2f6c97b8553e15c8687145676d589254f2be3308eef0a13982bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T02:23:32.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.9.service' 2024-03-28T02:23:32.932 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:23:32.932 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-28T02:23:32.932 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-28T02:23:32.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.10 2024-03-28T02:23:33.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:32 smithi153 systemd[1]: Stopping Ceph osd.10 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:23:33.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.154+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:33.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.154+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:33.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.154+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:33.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.154+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:33.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.154+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:33.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.154+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:33.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.154+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.862799+0000 front 2024-03-28T02:23:04.862949+0000 (oldest deadline 2024-03-28T02:23:30.162635+0000) 2024-03-28T02:23:33.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:33 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:33.074+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:33.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:33 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:33.074+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:33.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:33 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:33.074+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:33.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:33 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:33.074+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:33.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:33 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:33.074+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:33.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:33 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:33.074+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:33.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.247+0000 7fe66e6dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:23:33.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.247+0000 7fe66e6dc700 -1 osd.10 106 *** Got signal Terminated *** 2024-03-28T02:23:33.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:33 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:33.247+0000 7fe66e6dc700 -1 osd.10 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:23:34.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:34 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:34.064+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:34.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:34 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:34.064+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:34.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:34 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:34.064+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:34.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:34 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:34.064+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:34.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:34 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:34.064+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:34.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:34 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:34.064+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:34.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:34 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:34.162+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:34.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:34 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:34.162+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:34.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:34 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:34.162+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:34.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:34 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:34.162+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:34.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:34 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:34.162+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:34.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:34 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:34.162+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:34.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:34 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:34.162+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.862799+0000 front 2024-03-28T02:23:04.862949+0000 (oldest deadline 2024-03-28T02:23:30.162635+0000) 2024-03-28T02:23:35.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:35 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:35.073+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:35.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:35 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:35.073+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:35.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:35 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:35.073+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:35.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:35 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:35.073+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:35.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:35 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:35.073+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:35.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:35 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:35.073+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:35.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:35 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:35.073+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:35.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:35 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:35.196+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:35.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:35 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:35.196+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:35.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:35 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:35.196+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:35.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:35 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:35.196+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:35.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:35 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:35.196+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:35.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:35 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:35.196+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:35.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:35 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:35.196+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.862799+0000 front 2024-03-28T02:23:04.862949+0000 (oldest deadline 2024-03-28T02:23:30.162635+0000) 2024-03-28T02:23:36.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:36 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:36.080+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:36.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:36 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:36.080+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:36.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:36 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:36.080+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:36.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:36 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:36.080+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:36.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:36 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:36.080+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:36.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:36 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:36.080+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:36.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:36 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:36.080+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:36.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:36 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:36.244+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:36.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:36 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:36.244+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:36.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:36 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:36.244+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:36.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:36 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:36.244+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:36.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:36 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:36.244+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:36.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:36 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:36.244+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:36.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:36 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:36.244+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.862799+0000 front 2024-03-28T02:23:04.862949+0000 (oldest deadline 2024-03-28T02:23:30.162635+0000) 2024-03-28T02:23:37.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:37 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:37.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:37.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:37 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:37.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:37.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:37 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:37.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:37.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:37 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:37.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:37.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:37 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:37.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:37.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:37 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:37.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:37.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:37 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:37.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:37.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:37 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:37.239+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:15.954340+0000 front 2024-03-28T02:22:15.954390+0000 (oldest deadline 2024-03-28T02:22:41.254450+0000) 2024-03-28T02:23:37.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:37 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:37.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-28T02:22:25.756852+0000 front 2024-03-28T02:22:25.756906+0000 (oldest deadline 2024-03-28T02:22:51.657025+0000) 2024-03-28T02:23:37.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:37 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:37.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.358186+0000 front 2024-03-28T02:22:33.358241+0000 (oldest deadline 2024-03-28T02:22:58.058097+0000) 2024-03-28T02:23:37.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:37 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:37.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:40.859832+0000 front 2024-03-28T02:22:40.859868+0000 (oldest deadline 2024-03-28T02:23:03.759758+0000) 2024-03-28T02:23:37.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:37 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:37.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:49.660895+0000 front 2024-03-28T02:22:49.660877+0000 (oldest deadline 2024-03-28T02:23:11.360880+0000) 2024-03-28T02:23:37.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:37 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:37.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:56.661914+0000 front 2024-03-28T02:22:56.661886+0000 (oldest deadline 2024-03-28T02:23:20.161852+0000) 2024-03-28T02:23:37.735 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:37 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10[61457]: 2024-03-28T02:23:37.240+0000 7fe666cc8700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.862799+0000 front 2024-03-28T02:23:04.862949+0000 (oldest deadline 2024-03-28T02:23:30.162635+0000) 2024-03-28T02:23:38.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:38 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:38.081+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:38.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:38 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:38.081+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:38.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:38 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:38.081+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:38.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:38 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:38.081+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:38.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:38 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:38.081+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:38.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:38 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:38.081+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:38.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:38 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:38.081+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:38.926 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:38 smithi153 podman[73529]: 2024-03-28 02:23:38.633275893 +0000 UTC m=+5.507539304 container died ca71b92a28d7c2c1a0c9db7f3fa941902e67d33bf98700230cf8776b6d73fcaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:23:39.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:38 smithi153 podman[73529]: 2024-03-28 02:23:38.997209712 +0000 UTC m=+5.871473111 container cleanup ca71b92a28d7c2c1a0c9db7f3fa941902e67d33bf98700230cf8776b6d73fcaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=) 2024-03-28T02:23:39.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:38 smithi153 bash[73529]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10 2024-03-28T02:23:39.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:39 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:39.123+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:39.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:39 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:39.123+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:39.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:39 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:39.123+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:39.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:39 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:39.123+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:39.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:39 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:39.123+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:39.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:39 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:39.123+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:39.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:39 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:39.123+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:39.603 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:39 smithi153 podman[73546]: 2024-03-28 02:23:39.270689682 +0000 UTC m=+0.638998935 container remove ca71b92a28d7c2c1a0c9db7f3fa941902e67d33bf98700230cf8776b6d73fcaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-28T02:23:39.912 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:39 smithi153 podman[73638]: 2024-03-28 02:23:39.511294505 +0000 UTC m=+0.017739767 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:23:39.913 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:39 smithi153 podman[73638]: 2024-03-28 02:23:39.626635074 +0000 UTC m=+0.133080339 container create f9419ab0f2d2fe221b0856e52e466198335e081bc320c42db3e18dcacc6e1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T02:23:39.913 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:39 smithi153 podman[73638]: 2024-03-28 02:23:39.843331004 +0000 UTC m=+0.349776253 container init f9419ab0f2d2fe221b0856e52e466198335e081bc320c42db3e18dcacc6e1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-28T02:23:39.913 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:39 smithi153 podman[73638]: 2024-03-28 02:23:39.846318979 +0000 UTC m=+0.352764233 container start f9419ab0f2d2fe221b0856e52e466198335e081bc320c42db3e18dcacc6e1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T02:23:39.913 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:39 smithi153 podman[73638]: 2024-03-28 02:23:39.918254494 +0000 UTC m=+0.424699744 container attach f9419ab0f2d2fe221b0856e52e466198335e081bc320c42db3e18dcacc6e1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T02:23:40.235 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 28 02:23:40 smithi153 podman[73638]: 2024-03-28 02:23:40.131847579 +0000 UTC m=+0.638292834 container died f9419ab0f2d2fe221b0856e52e466198335e081bc320c42db3e18dcacc6e1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T02:23:40.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:40 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:40.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:40.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:40 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:40.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:40.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:40 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:40.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:40.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:40 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:40.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:40.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:40 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:40.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:40.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:40 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:40.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:40.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:40 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:40.095+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:40.743 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.10.service' 2024-03-28T02:23:40.784 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:23:40.784 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-28T02:23:40.784 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-28T02:23:40.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.11 2024-03-28T02:23:41.101 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:40 smithi178 systemd[1]: Stopping Ceph osd.11 for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:23:41.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.105+0000 7f305671b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:23:41.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.105+0000 7f305671b700 -1 osd.11 106 *** Got signal Terminated *** 2024-03-28T02:23:41.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.105+0000 7f305671b700 -1 osd.11 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T02:23:41.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.141+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:41.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.141+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:41.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.141+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:41.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.141+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:41.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.141+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:41.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.141+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:41.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:41 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:41.141+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:42.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:42 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:42.169+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:42.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:42 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:42.169+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:42.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:42 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:42.169+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:42.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:42 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:42.169+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:42.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:42 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:42.169+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:42.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:42 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:42.169+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:42.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:42 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:42.169+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:43.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:43 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:43.130+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:43.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:43 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:43.130+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:43.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:43 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:43.130+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:43.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:43 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:43.130+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:43.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:43 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:43.130+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:43.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:43 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:43.130+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:43.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:43 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:43.130+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:44.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:44.140+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:44.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:44.140+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:44.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:44.140+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:44.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:44.140+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:44.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:44.140+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:44.624 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:44.140+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:44.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:44.140+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:44.625 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:44 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:44.140+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6820 osd.8 since back 2024-03-28T02:23:21.695286+0000 front 2024-03-28T02:23:21.695317+0000 (oldest deadline 2024-03-28T02:23:43.295964+0000) 2024-03-28T02:23:45.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:45.113+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-28T02:22:16.783199+0000 front 2024-03-28T02:22:16.783367+0000 (oldest deadline 2024-03-28T02:22:40.883236+0000) 2024-03-28T02:23:45.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:45.113+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-28T02:22:33.086057+0000 front 2024-03-28T02:22:33.086087+0000 (oldest deadline 2024-03-28T02:22:56.586229+0000) 2024-03-28T02:23:45.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:45.113+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6814 osd.3 since back 2024-03-28T02:22:37.087097+0000 front 2024-03-28T02:22:37.087017+0000 (oldest deadline 2024-03-28T02:23:02.887639+0000) 2024-03-28T02:23:45.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:45.113+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-28T02:22:46.889401+0000 front 2024-03-28T02:22:46.889415+0000 (oldest deadline 2024-03-28T02:23:12.189239+0000) 2024-03-28T02:23:45.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:45.113+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-28T02:22:55.690294+0000 front 2024-03-28T02:22:55.690334+0000 (oldest deadline 2024-03-28T02:23:18.590460+0000) 2024-03-28T02:23:45.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:45.113+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.106:6822 osd.6 since back 2024-03-28T02:23:04.292885+0000 front 2024-03-28T02:23:04.292754+0000 (oldest deadline 2024-03-28T02:23:29.592758+0000) 2024-03-28T02:23:45.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:45.113+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-28T02:23:09.593272+0000 front 2024-03-28T02:23:09.593293+0000 (oldest deadline 2024-03-28T02:23:34.893214+0000) 2024-03-28T02:23:45.375 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:45 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11[62323]: 2024-03-28T02:23:45.113+0000 7f304ed07700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.178:6820 osd.8 since back 2024-03-28T02:23:21.695286+0000 front 2024-03-28T02:23:21.695317+0000 (oldest deadline 2024-03-28T02:23:43.295964+0000) 2024-03-28T02:23:46.676 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:46 smithi178 podman[73933]: 2024-03-28 02:23:46.421784068 +0000 UTC m=+5.440336208 container died e794eaf1f281b5d614a4964d13ed1904e7fbb01a353677744e543a2686f44f45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-28T02:23:47.044 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:46 smithi178 podman[73933]: 2024-03-28 02:23:46.780329156 +0000 UTC m=+5.798881303 container cleanup e794eaf1f281b5d614a4964d13ed1904e7fbb01a353677744e543a2686f44f45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-28T02:23:47.045 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:46 smithi178 bash[73933]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11 2024-03-28T02:23:47.374 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:47 smithi178 podman[73948]: 2024-03-28 02:23:47.050131329 +0000 UTC m=+0.622825436 container remove e794eaf1f281b5d614a4964d13ed1904e7fbb01a353677744e543a2686f44f45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T02:23:47.706 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:47 smithi178 podman[74041]: 2024-03-28 02:23:47.291674897 +0000 UTC m=+0.018674125 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T02:23:47.706 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:47 smithi178 podman[74041]: 2024-03-28 02:23:47.392641776 +0000 UTC m=+0.119641031 container create 3f16927c4011cfbf3043a2d3bd53a39b3ca01776bfa34b6b6d7a5f8eb381620d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T02:23:47.706 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:47 smithi178 podman[74041]: 2024-03-28 02:23:47.628594105 +0000 UTC m=+0.355593325 container init 3f16927c4011cfbf3043a2d3bd53a39b3ca01776bfa34b6b6d7a5f8eb381620d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T02:23:47.706 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:47 smithi178 podman[74041]: 2024-03-28 02:23:47.632406156 +0000 UTC m=+0.359405372 container start 3f16927c4011cfbf3043a2d3bd53a39b3ca01776bfa34b6b6d7a5f8eb381620d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T02:23:47.706 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:47 smithi178 podman[74041]: 2024-03-28 02:23:47.711412977 +0000 UTC m=+0.438412207 container attach 3f16927c4011cfbf3043a2d3bd53a39b3ca01776bfa34b6b6d7a5f8eb381620d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-28T02:23:48.124 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 28 02:23:47 smithi178 podman[74041]: 2024-03-28 02:23:47.922749256 +0000 UTC m=+0.649748482 container died 3f16927c4011cfbf3043a2d3bd53a39b3ca01776bfa34b6b6d7a5f8eb381620d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-osd-11-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T02:23:48.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@osd.11.service' 2024-03-28T02:23:48.538 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:23:48.538 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-28T02:23:48.538 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-28T02:23:48.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.a 2024-03-28T02:23:48.833 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:48 smithi106 systemd[1]: Stopping Ceph mds.a for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:23:49.084 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:48 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a[73998]: 2024-03-28T02:23:48.837+0000 7f2382866700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:23:49.085 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:48 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a[73998]: 2024-03-28T02:23:48.837+0000 7f2382866700 -1 mds.a *** got signal Terminated *** 2024-03-28T02:23:52.782 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:52 smithi106 podman[94097]: 2024-03-28 02:23:52.447128787 +0000 UTC m=+3.709406923 container died 26f52b2b47505a916bcc5e2cd0d8718fd3191479e8b9d190babe9beee450ea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T02:23:53.058 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:52 smithi106 podman[94097]: 2024-03-28 02:23:52.862489354 +0000 UTC m=+4.124767531 container cleanup 26f52b2b47505a916bcc5e2cd0d8718fd3191479e8b9d190babe9beee450ea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T02:23:53.058 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:52 smithi106 bash[94097]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a 2024-03-28T02:23:53.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.a.service' 2024-03-28T02:23:53.310 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:53 smithi106 podman[94111]: 2024-03-28 02:23:53.063814705 +0000 UTC m=+0.616945989 container remove 26f52b2b47505a916bcc5e2cd0d8718fd3191479e8b9d190babe9beee450ea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-a, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T02:23:53.310 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:53 smithi106 systemd[1]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.a.service: Deactivated successfully. 2024-03-28T02:23:53.310 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:53 smithi106 systemd[1]: Stopped Ceph mds.a for d222a03c-eca6-11ee-95d0-87774f69a715. 2024-03-28T02:23:53.310 INFO:journalctl@ceph.mds.a.smithi106.stdout:Mar 28 02:23:53 smithi106 systemd[1]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.a.service: Consumed 2.062s CPU time. 2024-03-28T02:23:53.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:23:53.321 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-28T02:23:53.321 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-28T02:23:53.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.d 2024-03-28T02:23:53.584 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:23:53 smithi106 systemd[1]: Stopping Ceph mds.d for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:23:54.085 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:23:53 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-d[73044]: 2024-03-28T02:23:53.640+0000 7fdaccd28700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:23:54.085 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:23:53 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-d[73044]: 2024-03-28T02:23:53.640+0000 7fdaccd28700 -1 mds.d *** got signal Terminated *** 2024-03-28T02:23:55.970 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:23:55 smithi106 podman[94219]: 2024-03-28 02:23:55.701518239 +0000 UTC m=+2.155614642 container died b97c63748bd24a9b0cf78e315a2d7b87df398d9a57d865614ab61099e93955e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T02:23:56.320 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:23:56 smithi106 podman[94219]: 2024-03-28 02:23:56.058008585 +0000 UTC m=+2.512105022 container cleanup b97c63748bd24a9b0cf78e315a2d7b87df398d9a57d865614ab61099e93955e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T02:23:56.320 INFO:journalctl@ceph.mds.d.smithi106.stdout:Mar 28 02:23:56 smithi106 bash[94219]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-d 2024-03-28T02:23:56.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.d.service' 2024-03-28T02:23:56.510 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:23:56.511 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-28T02:23:56.511 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-28T02:23:56.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.g 2024-03-28T02:23:56.835 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:23:56 smithi106 systemd[1]: Stopping Ceph mds.g for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:23:56.835 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:23:56 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g[73652]: 2024-03-28T02:23:56.827+0000 7f58c769d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:23:56.835 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:23:56 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g[73652]: 2024-03-28T02:23:56.827+0000 7f58c769d700 -1 mds.g *** got signal Terminated *** 2024-03-28T02:24:00.715 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:24:00 smithi106 podman[94341]: 2024-03-28 02:24:00.443216712 +0000 UTC m=+3.705410667 container died bce84fdd6b88f97c9e845c696d07bf2ab888f809d60e4aae14bf261aff56fef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T02:24:01.057 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:24:00 smithi106 podman[94341]: 2024-03-28 02:24:00.819785251 +0000 UTC m=+4.081979217 container cleanup bce84fdd6b88f97c9e845c696d07bf2ab888f809d60e4aae14bf261aff56fef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T02:24:01.057 INFO:journalctl@ceph.mds.g.smithi106.stdout:Mar 28 02:24:00 smithi106 bash[94341]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-g 2024-03-28T02:24:01.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.g.service' 2024-03-28T02:24:01.215 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:01.216 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-28T02:24:01.216 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-28T02:24:01.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.j 2024-03-28T02:24:01.334 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:24:01 smithi106 systemd[1]: Stopping Ceph mds.j for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:24:01.835 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:24:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-j[72658]: 2024-03-28T02:24:01.532+0000 7f3b081c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:24:01.835 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:24:01 smithi106 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-j[72658]: 2024-03-28T02:24:01.532+0000 7f3b081c8700 -1 mds.j *** got signal Terminated *** 2024-03-28T02:24:05.144 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:24:04 smithi106 podman[94464]: 2024-03-28 02:24:04.842178153 +0000 UTC m=+3.405457431 container died a89b9033c742fbc24f5212ba9f70bb9ebcf4a079c382327cd7c2da132681ba33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-28T02:24:05.435 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:24:05 smithi106 podman[94464]: 2024-03-28 02:24:05.248250801 +0000 UTC m=+3.811530081 container cleanup a89b9033c742fbc24f5212ba9f70bb9ebcf4a079c382327cd7c2da132681ba33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T02:24:05.436 INFO:journalctl@ceph.mds.j.smithi106.stdout:Mar 28 02:24:05 smithi106 bash[94464]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-j 2024-03-28T02:24:05.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.j.service' 2024-03-28T02:24:05.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:05.596 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-28T02:24:05.596 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-28T02:24:05.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.b 2024-03-28T02:24:05.968 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:24:05 smithi153 systemd[1]: Stopping Ceph mds.b for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:24:06.235 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:24:05 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b[67179]: 2024-03-28T02:24:05.972+0000 7f6302603700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:24:06.235 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:24:05 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b[67179]: 2024-03-28T02:24:05.972+0000 7f6302603700 -1 mds.b *** got signal Terminated *** 2024-03-28T02:24:08.727 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:24:08 smithi153 podman[73757]: 2024-03-28 02:24:08.452110886 +0000 UTC m=+2.658396638 container died 693b245038b96d31220ae81b52ec10c932fbec99d214619fb366808bb9cb6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:24:08.985 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:24:08 smithi153 podman[73757]: 2024-03-28 02:24:08.831051197 +0000 UTC m=+3.037336907 container cleanup 693b245038b96d31220ae81b52ec10c932fbec99d214619fb366808bb9cb6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T02:24:08.985 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 28 02:24:08 smithi153 bash[73757]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-b 2024-03-28T02:24:09.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.b.service' 2024-03-28T02:24:09.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:09.288 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-28T02:24:09.288 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-28T02:24:09.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.e 2024-03-28T02:24:09.623 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:24:09 smithi153 systemd[1]: Stopping Ceph mds.e for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:24:09.985 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:24:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e[66549]: 2024-03-28T02:24:09.628+0000 7f5f2d919700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:24:09.985 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:24:09 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e[66549]: 2024-03-28T02:24:09.628+0000 7f5f2d919700 -1 mds.e *** got signal Terminated *** 2024-03-28T02:24:12.717 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:24:12 smithi153 podman[73879]: 2024-03-28 02:24:12.466991546 +0000 UTC m=+2.948961441 container died a4218b5047f29f41a14e23aece7672d81f9cfaa951ab012c145d88cb886eef44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T02:24:12.985 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:24:12 smithi153 podman[73879]: 2024-03-28 02:24:12.820684972 +0000 UTC m=+3.302654863 container cleanup a4218b5047f29f41a14e23aece7672d81f9cfaa951ab012c145d88cb886eef44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T02:24:12.985 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 28 02:24:12 smithi153 bash[73879]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-e 2024-03-28T02:24:13.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.e.service' 2024-03-28T02:24:13.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:13.268 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-28T02:24:13.268 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-28T02:24:13.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.h 2024-03-28T02:24:13.735 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:24:13 smithi153 systemd[1]: Stopping Ceph mds.h for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:24:13.735 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:24:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h[66868]: 2024-03-28T02:24:13.590+0000 7f3cac504700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:24:13.735 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:24:13 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h[66868]: 2024-03-28T02:24:13.590+0000 7f3cac504700 -1 mds.h *** got signal Terminated *** 2024-03-28T02:24:16.736 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:24:16 smithi153 podman[74013]: 2024-03-28 02:24:16.46943861 +0000 UTC m=+2.983403921 container died 3ebf9fb3013b026a66ec2cb23f2f517c10ba802577b61f011e23005d56639808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-28T02:24:17.079 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:24:16 smithi153 podman[74013]: 2024-03-28 02:24:16.874549379 +0000 UTC m=+3.388514684 container cleanup 3ebf9fb3013b026a66ec2cb23f2f517c10ba802577b61f011e23005d56639808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T02:24:17.079 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 28 02:24:16 smithi153 bash[74013]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-h 2024-03-28T02:24:17.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.h.service' 2024-03-28T02:24:17.253 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:17.253 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-28T02:24:17.254 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-28T02:24:17.254 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.k 2024-03-28T02:24:17.736 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:24:17 smithi153 systemd[1]: Stopping Ceph mds.k for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:24:17.736 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:24:17 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k[66226]: 2024-03-28T02:24:17.577+0000 7f1be3d4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:24:17.736 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:24:17 smithi153 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k[66226]: 2024-03-28T02:24:17.577+0000 7f1be3d4d700 -1 mds.k *** got signal Terminated *** 2024-03-28T02:24:20.735 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:24:20 smithi153 podman[74134]: 2024-03-28 02:24:20.456561965 +0000 UTC m=+2.984671103 container died 87cffb547bf8bdcb04087409f83d88ecd9fb85041d07f6a1ab53f0a639953292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:24:21.047 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:24:20 smithi153 podman[74134]: 2024-03-28 02:24:20.761337408 +0000 UTC m=+3.289446538 container cleanup 87cffb547bf8bdcb04087409f83d88ecd9fb85041d07f6a1ab53f0a639953292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T02:24:21.048 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:24:20 smithi153 bash[74134]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k 2024-03-28T02:24:21.048 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 28 02:24:20 smithi153 podman[74148]: 2024-03-28 02:24:20.97029918 +0000 UTC m=+0.514519636 container remove 87cffb547bf8bdcb04087409f83d88ecd9fb85041d07f6a1ab53f0a639953292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-k, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T02:24:21.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.k.service' 2024-03-28T02:24:21.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:21.145 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-28T02:24:21.146 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-28T02:24:21.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.c 2024-03-28T02:24:21.467 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:24:21 smithi178 systemd[1]: Stopping Ceph mds.c for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:24:21.744 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:24:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c[67347]: 2024-03-28T02:24:21.471+0000 7f93e0e5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:24:21.744 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:24:21 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c[67347]: 2024-03-28T02:24:21.471+0000 7f93e0e5c700 -1 mds.c *** got signal Terminated *** 2024-03-28T02:24:24.775 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:24:24 smithi178 podman[74148]: 2024-03-28 02:24:24.457778843 +0000 UTC m=+3.109460934 container died a4878635d498d49025145ea64ef21042e34a71fb986aa60f506924128455837c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T02:24:25.092 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:24:24 smithi178 podman[74148]: 2024-03-28 02:24:24.86252588 +0000 UTC m=+3.514207976 container cleanup a4878635d498d49025145ea64ef21042e34a71fb986aa60f506924128455837c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T02:24:25.092 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 28 02:24:24 smithi178 bash[74148]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-c 2024-03-28T02:24:25.207 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.c.service' 2024-03-28T02:24:25.245 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:25.245 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-28T02:24:25.246 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-28T02:24:25.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.f 2024-03-28T02:24:25.374 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:24:25 smithi178 systemd[1]: Stopping Ceph mds.f for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:24:25.874 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:24:25 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f[67037]: 2024-03-28T02:24:25.565+0000 7f1eff972700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:24:25.874 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:24:25 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f[67037]: 2024-03-28T02:24:25.565+0000 7f1eff972700 -1 mds.f *** got signal Terminated *** 2024-03-28T02:24:28.728 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:24:28 smithi178 podman[74281]: 2024-03-28 02:24:28.457035804 +0000 UTC m=+2.991556301 container died e226f5b558f622d4981ef8bb458761dc4d94ca7bcda8cad49bba49b194fc9f8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T02:24:29.078 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:24:28 smithi178 podman[74281]: 2024-03-28 02:24:28.81571974 +0000 UTC m=+3.350240253 container cleanup e226f5b558f622d4981ef8bb458761dc4d94ca7bcda8cad49bba49b194fc9f8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T02:24:29.078 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 28 02:24:28 smithi178 bash[74281]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-f 2024-03-28T02:24:29.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.f.service' 2024-03-28T02:24:29.265 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:29.266 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-28T02:24:29.266 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-28T02:24:29.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.i 2024-03-28T02:24:29.589 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:24:29 smithi178 systemd[1]: Stopping Ceph mds.i for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:24:29.874 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:24:29 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i[67656]: 2024-03-28T02:24:29.594+0000 7f2cd8fff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:24:29.874 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:24:29 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i[67656]: 2024-03-28T02:24:29.594+0000 7f2cd8fff700 -1 mds.i *** got signal Terminated *** 2024-03-28T02:24:32.866 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:24:32 smithi178 podman[74402]: 2024-03-28 02:24:32.462148308 +0000 UTC m=+2.976038287 container died 5eb0e9cd985a7e6cb0675da502ef528f1d7023c0e2d1d2190055f5ae211f20b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T02:24:33.125 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:24:32 smithi178 podman[74402]: 2024-03-28 02:24:32.952520057 +0000 UTC m=+3.466410021 container cleanup 5eb0e9cd985a7e6cb0675da502ef528f1d7023c0e2d1d2190055f5ae211f20b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T02:24:33.125 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 28 02:24:32 smithi178 bash[74402]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-i 2024-03-28T02:24:33.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.i.service' 2024-03-28T02:24:33.335 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:33.335 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-28T02:24:33.335 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-28T02:24:33.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.l 2024-03-28T02:24:33.625 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:24:33 smithi178 systemd[1]: Stopping Ceph mds.l for d222a03c-eca6-11ee-95d0-87774f69a715... 2024-03-28T02:24:34.125 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:24:33 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l[66730]: 2024-03-28T02:24:33.660+0000 7f3a2455f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T02:24:34.125 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:24:33 smithi178 ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l[66730]: 2024-03-28T02:24:33.660+0000 7f3a2455f700 -1 mds.l *** got signal Terminated *** 2024-03-28T02:24:36.778 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:24:36 smithi178 podman[74525]: 2024-03-28 02:24:36.459253194 +0000 UTC m=+2.906242945 container died d8f3d10520669311aff7ebb6586234acfe4fe2e5698c9a3936249e09ddc782b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T02:24:37.069 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:24:36 smithi178 podman[74525]: 2024-03-28 02:24:36.880923025 +0000 UTC m=+3.327912769 container cleanup d8f3d10520669311aff7ebb6586234acfe4fe2e5698c9a3936249e09ddc782b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-28T02:24:37.069 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 28 02:24:36 smithi178 bash[74525]: ceph-d222a03c-eca6-11ee-95d0-87774f69a715-mds-l 2024-03-28T02:24:37.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d222a03c-eca6-11ee-95d0-87774f69a715@mds.l.service' 2024-03-28T02:24:37.324 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T02:24:37.325 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-28T02:24:37.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d222a03c-eca6-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T02:24:37.519 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:24:39.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d222a03c-eca6-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T02:24:40.158 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:24:42.762 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d222a03c-eca6-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T02:24:42.952 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:24:45.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T02:24:45.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T02:24:45.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T02:24:45.413 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-28T02:24:45.415 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/remote/smithi106/crash 2024-03-28T02:24:45.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/crash -- . 2024-03-28T02:24:45.459 INFO:teuthology.orchestra.run.smithi106.stderr:tar: /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T02:24:45.459 INFO:teuthology.orchestra.run.smithi106.stderr:tar: Error is not recoverable: exiting now 2024-03-28T02:24:45.461 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/remote/smithi153/crash 2024-03-28T02:24:45.462 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/crash -- . 2024-03-28T02:24:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T02:24:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-03-28T02:24:45.504 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/remote/smithi178/crash 2024-03-28T02:24:45.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/crash -- . 2024-03-28T02:24:45.545 INFO:teuthology.orchestra.run.smithi178.stderr:tar: /var/lib/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T02:24:45.545 INFO:teuthology.orchestra.run.smithi178.stderr:tar: Error is not recoverable: exiting now 2024-03-28T02:24:45.546 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-28T02:24:45.547 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T02:24:45.580 INFO:teuthology.orchestra.run.smithi106.stdout:2024-03-28T02:04:35.172964+0000 mon.a (mon.0) 475 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T02:24:45.580 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-28T02:24:45.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[SEC\]' /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T02:24:45.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[ERR\]' /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T02:24:45.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[WRN\]' /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T02:24:45.799 INFO:teuthology.orchestra.run.smithi106.stdout:2024-03-28T02:04:35.172964+0000 mon.a (mon.0) 475 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T02:24:45.799 INFO:tasks.cephadm:Compressing logs... 2024-03-28T02:24:45.799 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T02:24:45.842 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T02:24:45.845 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T02:24:45.866 INFO:teuthology.orchestra.run.smithi106.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T02:24:45.867 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T02:24:45.867 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-28T02:24:45.867 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-28T02:24:45.867 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T02:24:45.867 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-28T02:24:45.868 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-28T02:24:45.868 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-28T02:24:45.868 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-28T02:24:45.868 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-28T02:24:45.871 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mgr.x.log: 90.3% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T02:24:45.871 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T02:24:45.872 INFO:teuthology.orchestra.run.smithi178.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T02:24:45.872 INFO:teuthology.orchestra.run.smithi178.stderr:: No such file or directory 2024-03-28T02:24:45.872 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-28T02:24:45.872 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T02:24:45.872 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.l.log: -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-28T02:24:45.872 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-28T02:24:45.878 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-28T02:24:45.878 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.i.log: /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.11.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T02:24:45.878 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-28T02:24:45.878 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-28T02:24:45.878 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mgr.z.log: /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-28T02:24:45.879 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T02:24:45.879 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.f.log: 90.4% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T02:24:45.879 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log 2024-03-28T02:24:45.879 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.7% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T02:24:45.879 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-28T02:24:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T02:24:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-03-28T02:24:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-28T02:24:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.4.log: /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.7.log: 90.4% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T02:24:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-28T02:24:45.882 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.0.log: 94.7% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-28T02:24:45.882 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log 2024-03-28T02:24:45.883 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.cephadm.log: 84.1% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T02:24:45.883 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T02:24:45.883 INFO:teuthology.orchestra.run.smithi106.stderr: /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T02:24:45.883 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-28T02:24:45.883 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-volume.log: 87.5% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T02:24:45.883 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-28T02:24:45.885 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log: 87.5% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T02:24:45.885 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-28T02:24:45.885 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-28T02:24:45.885 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T02:24:45.890 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.9.log: 91.0% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T02:24:45.891 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-28T02:24:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T02:24:45.894 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-28T02:24:45.896 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-28T02:24:45.896 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-28T02:24:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log 2024-03-28T02:24:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T02:24:45.911 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T02:24:45.913 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log: 91.8% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-28T02:24:45.915 INFO:teuthology.orchestra.run.smithi153.stderr: 87.5% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T02:24:45.916 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-28T02:24:45.921 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T02:24:45.927 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-28T02:24:45.929 INFO:teuthology.orchestra.run.smithi178.stderr: 93.2% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-28T02:24:45.931 INFO:teuthology.orchestra.run.smithi106.stderr: 94.5% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-28T02:24:45.934 INFO:teuthology.orchestra.run.smithi178.stderr: 93.1% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-28T02:24:45.943 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mon.b.log: 94.5% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-28T02:24:45.944 INFO:teuthology.orchestra.run.smithi178.stderr: 93.0% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.i.log.gz 2024-03-28T02:24:45.945 INFO:teuthology.orchestra.run.smithi178.stderr: 94.7% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-28T02:24:45.954 INFO:teuthology.orchestra.run.smithi153.stderr: 90.9% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T02:24:45.957 INFO:teuthology.orchestra.run.smithi153.stderr: 93.4% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-28T02:24:45.964 INFO:teuthology.orchestra.run.smithi106.stderr: 94.5% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-28T02:24:46.156 INFO:teuthology.orchestra.run.smithi153.stderr: 96.4% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-28T02:24:46.194 INFO:teuthology.orchestra.run.smithi106.stderr: 89.8% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-28T02:24:46.204 INFO:teuthology.orchestra.run.smithi153.stderr: 91.9% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-28T02:24:46.283 INFO:teuthology.orchestra.run.smithi178.stderr: 92.0% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-28T02:24:47.084 INFO:teuthology.orchestra.run.smithi106.stderr: 90.4% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-28T02:24:48.407 INFO:teuthology.orchestra.run.smithi153.stderr: 94.8% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-28T02:24:49.960 INFO:teuthology.orchestra.run.smithi178.stderr: 92.9% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-28T02:24:50.170 INFO:teuthology.orchestra.run.smithi178.stderr: 92.5% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-28T02:24:50.202 INFO:teuthology.orchestra.run.smithi106.stderr: 92.8% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-28T02:24:50.310 INFO:teuthology.orchestra.run.smithi153.stderr: 92.7% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-28T02:24:50.861 INFO:teuthology.orchestra.run.smithi153.stderr: 92.9% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-28T02:24:51.224 INFO:teuthology.orchestra.run.smithi106.stderr: 92.7% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-28T02:24:51.421 INFO:teuthology.orchestra.run.smithi106.stderr: 92.8% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-28T02:24:51.822 INFO:teuthology.orchestra.run.smithi178.stderr: 92.6% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-28T02:24:52.394 INFO:teuthology.orchestra.run.smithi153.stderr: 92.6% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-28T02:24:52.480 INFO:teuthology.orchestra.run.smithi153.stderr: 92.5% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-28T02:24:52.482 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-28T02:24:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m6.623s 2024-03-28T02:24:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m24.915s 2024-03-28T02:24:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.836s 2024-03-28T02:24:52.798 INFO:teuthology.orchestra.run.smithi106.stderr: 92.6% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-28T02:24:52.800 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-03-28T02:24:52.800 INFO:teuthology.orchestra.run.smithi106.stderr:real 0m6.946s 2024-03-28T02:24:52.800 INFO:teuthology.orchestra.run.smithi106.stderr:user 0m23.045s 2024-03-28T02:24:52.800 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m0.816s 2024-03-28T02:24:53.480 INFO:teuthology.orchestra.run.smithi178.stderr: 92.5% -- replaced with /var/log/ceph/d222a03c-eca6-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-28T02:24:53.482 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-28T02:24:53.482 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m7.623s 2024-03-28T02:24:53.482 INFO:teuthology.orchestra.run.smithi178.stderr:user 0m21.753s 2024-03-28T02:24:53.485 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m0.794s 2024-03-28T02:24:53.485 INFO:tasks.cephadm:Archiving logs... 2024-03-28T02:24:53.485 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/remote/smithi106/log 2024-03-28T02:24:53.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T02:24:57.055 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/remote/smithi153/log 2024-03-28T02:24:57.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T02:25:00.852 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/remote/smithi178/log 2024-03-28T02:25:00.853 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T02:25:04.581 INFO:tasks.cephadm:Removing cluster... 2024-03-28T02:25:04.582 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d222a03c-eca6-11ee-95d0-87774f69a715 --force 2024-03-28T02:25:04.772 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:25:05.054 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d222a03c-eca6-11ee-95d0-87774f69a715 --force 2024-03-28T02:25:05.246 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:25:05.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d222a03c-eca6-11ee-95d0-87774f69a715 --force 2024-03-28T02:25:05.728 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: d222a03c-eca6-11ee-95d0-87774f69a715 2024-03-28T02:25:06.025 INFO:tasks.cephadm:Removing cephadm ... 2024-03-28T02:25:06.025 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T02:25:06.043 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T02:25:06.060 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T02:25:06.075 INFO:tasks.cephadm:Teardown complete 2024-03-28T02:25:06.075 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T02:25:06.086 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T02:25:06.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T02:25:06.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T02:25:06.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T02:25:06.208 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T02:25:06.218 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T02:25:06.219 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T02:25:06.221 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T02:25:06.223 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T02:25:06.235 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-03-28T02:25:06.235 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-28T02:25:06.237 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-03-28T02:25:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T02:25:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-03-28T02:25:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +1056us[+1056us] +/- 77ms 2024-03-28T02:25:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -1907us[-1890us] +/- 59ms 2024-03-28T02:25:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +1701us[+1718us] +/- 86ms 2024-03-28T02:25:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T02:25:06.240 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T02:25:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-28T02:25:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +1043us[+1057us] +/- 77ms 2024-03-28T02:25:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -1926us[-1912us] +/- 59ms 2024-03-28T02:25:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +1681us[+1681us] +/- 86ms 2024-03-28T02:25:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T02:25:06.242 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T02:25:06.242 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-03-28T02:25:06.242 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +1048us[+1048us] +/- 77ms 2024-03-28T02:25:06.242 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -1892us[-1912us] +/- 59ms 2024-03-28T02:25:06.242 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +1691us[+1672us] +/- 86ms 2024-03-28T02:25:06.242 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T02:25:06.242 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T02:25:06.252 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T02:25:06.253 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T02:25:06.273 DEBUG:teuthology.orchestra.run.smithi106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T02:25:06.366 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T02:25:06.440 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T02:25:06.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T02:25:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T02:25:06.542 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T02:25:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T02:25:06.572 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T02:25:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T02:25:06.602 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T02:25:06.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T02:25:06.648 INFO:teuthology.task.internal:Duration was 2063.237550 seconds 2024-03-28T02:25:06.649 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T02:25:06.681 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T02:25:06.682 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T02:25:06.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T02:25:06.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T02:25:06.727 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T02:25:06.727 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T02:25:06.730 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T02:25:07.155 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T02:25:07.155 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-03-28T02:25:07.155 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T02:25:07.177 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-03-28T02:25:07.178 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T02:25:07.205 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-03-28T02:25:07.205 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T02:25:07.237 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T02:25:07.237 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T02:25:07.240 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T02:25:07.248 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T02:25:07.305 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T02:25:07.305 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T02:25:07.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T02:25:07.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T02:25:07.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T02:25:07.718 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T02:25:07.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T02:25:07.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T02:25:07.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T02:25:07.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T02:25:07.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T02:25:07.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T02:25:07.814 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T02:25:07.835 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-03-28T02:25:07.837 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-03-28T02:25:07.864 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-03-28T02:25:07.889 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T02:25:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T02:25:07.904 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T02:25:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T02:25:07.918 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T02:25:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T02:25:07.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T02:25:07.954 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T02:25:07.955 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/remote/smithi106 2024-03-28T02:25:07.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T02:25:08.001 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/remote/smithi153 2024-03-28T02:25:08.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T02:25:08.039 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289/remote/smithi178 2024-03-28T02:25:08.039 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T02:25:08.078 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T02:25:08.078 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T02:25:08.080 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T02:25:08.082 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T02:25:08.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T02:25:08.139 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T02:25:08.139 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T02:25:08.158 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T02:25:08.158 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T02:25:08.160 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T02:25:08.162 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T02:25:08.172 INFO:teuthology.orchestra.run.smithi106.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 02:25 /home/ubuntu/cephtest 2024-03-28T02:25:08.174 INFO:teuthology.orchestra.run.smithi153.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 02:25 /home/ubuntu/cephtest 2024-03-28T02:25:08.181 INFO:teuthology.orchestra.run.smithi178.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 02:25 /home/ubuntu/cephtest 2024-03-28T02:25:08.182 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T02:25:08.192 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T02:25:08.258 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T02:25:08.283 DEBUG:teuthology.nuke:shortname: smithi106 2024-03-28T02:25:08.283 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T02:25:08.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 01:42:03.260581', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDNAGKdEIkOeJdGKSc6vvM9vhCHIa6CwM6nCt7DBJnb28/FZeHbZP9ERQHmy2FA4MqdvMsXR7fXwHh5j9ytxrDE='} 2024-03-28T02:25:08.309 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-28T02:25:08.309 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T02:25:08.329 DEBUG:teuthology.nuke:shortname: smithi178 2024-03-28T02:25:08.329 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T02:25:08.332 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 01:42:03.257150', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHsnuOuuYtDHnkI1Br+l96AJYXcu28EuymJYWEf70mN7nvEIJGYBgF46BbrQoI+7RYW/vDw6TpVGSNuerNtMcw='} 2024-03-28T02:25:08.362 INFO:teuthology.orchestra.console.smithi106:Power off 2024-03-28T02:25:08.363 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T02:25:08.371 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625289', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 01:42:03.258961', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOJYoPZDfxS+Y1BtjGEiDw4H4FGd9SCJqVutLJhWFCSVGatw8gGpssdXLA9drWNqQ0OFhvs3Aq0mc7tUbtlSak='} 2024-03-28T02:25:08.385 DEBUG:teuthology.orchestra.console.smithi106:power off output: Chassis Power Control: Down/Off 2024-03-28T02:25:08.386 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:08.408 INFO:teuthology.orchestra.console.smithi153:Power off 2024-03-28T02:25:08.408 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T02:25:08.418 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-28T02:25:08.431 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-03-28T02:25:08.432 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:08.445 INFO:teuthology.orchestra.console.smithi178:Power off 2024-03-28T02:25:08.446 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T02:25:08.466 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2024-03-28T02:25:08.467 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:08.477 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-28T02:25:08.488 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-03-28T02:25:12.419 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:12.478 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:12.489 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:12.530 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-28T02:25:12.587 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-28T02:25:12.600 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-03-28T02:25:16.534 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:16.588 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:16.601 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:16.645 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-28T02:25:16.696 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-03-28T02:25:16.711 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-03-28T02:25:16.797 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-03-28T02:25:20.650 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:20.712 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T02:25:20.761 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is off 2024-03-28T02:25:20.822 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2024-03-28T02:25:20.862 INFO:teuthology.orchestra.console.smithi106:Power off completed 2024-03-28T02:25:20.922 INFO:teuthology.orchestra.console.smithi178:Power off completed 2024-03-28T02:25:21.027 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} duration: 2063.237550497055 failure_reason: '"2024-03-28T02:04:35.172964+0000 mon.a (mon.0) 475 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log' flavor: default owner: mchangir success: false 2024-03-28T02:25:21.027 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T02:25:21.116 INFO:teuthology.run:FAIL